<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eduardo Alvarez on Medium]]></title>
        <description><![CDATA[Stories by Eduardo Alvarez on Medium]]></description>
        <link>https://medium.com/@eduand-alvarez?source=rss-e49cc416a8ef------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*O5m2-s0IdSz2md3JkBAOow.png</url>
            <title>Stories by Eduardo Alvarez on Medium</title>
            <link>https://medium.com/@eduand-alvarez?source=rss-e49cc416a8ef------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:17:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eduand-alvarez/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[5 Agentic Design Patterns Everyone Should Know]]></title>
            <link>https://eduand-alvarez.medium.com/5-agentic-design-patterns-everyone-should-know-6825cd989178?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/6825cd989178</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Mon, 11 May 2026 12:36:57 GMT</pubDate>
            <atom:updated>2026-05-11T12:36:57.051Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R5sIEbABIW8F6sIMBda-mg.png" /></figure><h4>A soft introduction to basic design patterns that underpin many of today’s agentic systems</h4><p>Agentic AI is clearly the next major scaling law for the industry. Despite being around for years, we are only now seeing model performance reach the critical thresholds that allow this technology to cross over from research to practical utility at scale.</p><p>Since best practices are still materializing, I’ve always found that a basic understanding of simple design patterns is the best place to start. At their foundation, these patterns provide structure based on commonalities shared across proven implementations — serving as the building blocks for understanding the core interactions between agents, users, models, and tools.</p><p>In this blog, I will simply introduce 5 common design patterns:</p><ol><li>Single Agent</li><li>Sequential Agents</li><li>Parallel Agents</li><li>Loop &amp; Critic</li><li>Coordinator &amp; Sub-agents<br><em>extra: Sub-Agents as Tools</em></li></ol><h3>1. Single Agent</h3><p>The single agent pattern (Figure 1) is, by nature, the simplest to implement. It requires the management of a narrower set of system-level instructions and tool-definition assets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2A5Z1rLkDpq30kp2rWm81Q.png" /><figcaption>Figure 1. Single agent design pattern diagram</figcaption></figure><p>This pattern requires giving a single agent access to relevant tool definitions and instructions. For simpler tasks requiring just a few specialized tools, this pattern fares quite well, benefiting from:</p><ul><li>Ease of deployment</li><li>Simpler debugging</li><li>Lower end-to-end task latency</li></ul><p>Where the single agent pattern begins to show weaknesses is with challenging tasks that require the orchestration of complex multi-step logic and many tools. These challenges could be summarized as:</p><ul><li>Difficulty following large system prompt logic</li><li>Becomes a single point of failure for entire workflow</li></ul><h3>2. Sequential Agents</h3><p>Sequential agents are one of the simplest instantiations of a multi-agent system. They require that agents behave like an assembly line — where each preceding agent’s output is the direct input into the subsequent agent. For simplicity, we consider a sequential agent pattern composed of two specialized agents (Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*avCKV6-BmTpbe3Sa6Aszrw.png" /><figcaption>Figure 2. Sequential agent design pattern diagram</figcaption></figure><p>Sequential agents are able to transfer data through a shared short-term memory state. This allows both agents to access not only data, but also the state of the task as they share responsibilities across the overall workflow.</p><p><strong>Specialized agents have the advantage of:</strong></p><ul><li>A narrower set of specialized instructions, allowing them to dedicate their context to perfoming sub-tasks</li><li>The sequential aspect also helps “hard code” sequence logic into the overall workflow — eliminating the non-deterministic risk of out-of-order execution that often leads to failure in multi-step logic tasks</li><li>Fewer LLM calls per agent, enabling easier debugging and state management</li></ul><p>A clear downside of sequential agents is the rigidity and rules that bound their execution:</p><ul><li>Steps in the sequence typically can’t be skipped, even if they are not needed for a specific variation of the workflow</li><li>Latency accumulation that results from agents waiting for their required inputs from a previous agent</li></ul><h3>3. Parallel Agents</h3><p>Some tasks benefit from multiple agents but don’t require sequential execution — this is where parallel agent patterns (Figure 3) can be quite beneficial. The execution of independent tasks in parallel has significant end-to-end latency benefits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9qXtgdALxKEll4sMS5MBw.png" /><figcaption>Figure 3. Parallel agent design pattern diagram</figcaption></figure><p>Like with any parallel tasks in computing, some risks are introduced:</p><ul><li>Higher per-unit-time resource spend as parallel agents simultaneously consume tokens and compete for resources</li><li>Harder debugging and root-cause analysis</li><li>Race conditions, where the outputs of the parallel workflows yield incompatible results</li></ul><p>Race conditions are particularly costly, since they can force full round-trip execution of the entire workflow. To reduce this risk, parallel agent patterns are often paired with a final aggregator agent. The aggregator can be thought of as a final sequential agent that is in charge of consolidating the outputs of the parallel agents and producing an output that is cohesive and aligned with the parameters of the request.</p><h3>4. Loop &amp; Critic</h3><p>Imagine that an agent’s output needs to meet a strict requirement. For example, the output of a coding agent must be able to pass a specific unit test before being considered valid. The loop and critic patter, provides this simple yet powerful guardrail technique where a specialized agent serves as a “critic” or “approver” of another agent’s output (Figure 4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQqqEQ7pj0KoN3-UcYZezg.png" /><figcaption>Figure 4. Loop &amp; Critic design patter diagram. Note, the model and tool boxes have been abstracted away for the sake of simplifying this visual.</figcaption></figure><p>The main pitfall of this pattern is the risk of infinite retry loops that necessitate:</p><ul><li>Configuration of maximum loop logic to prevent vicious and wasteful rejection/retry loops between the critic and producer</li><li>Careful exit-criteria definitions to ensure fewer false-negative rejections and false-positive passes by the critic</li></ul><h3>5. Coordinator &amp; Sub-Agents</h3><p>Probably the most common pattern today, the coordinator pattern (Figure 5) introduces agent hierarchy and task decomposition — which allows a main agent to coordinate a workflow and manage the state of multiple sub-agents. The sub-agents in this pattern often have predefined special instructions and tool definitions, and it is the main agent’s job to assign and track the work performed by the sub-agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SpA6qOU1BoHMkxafPPpYdA.png" /><figcaption>Figure 5. Coordinator design pattern diagram. Note, the model and tool boxes have been abstracted away for the sake of simplifying this visual.</figcaption></figure><p>Sub-agents in this pattern can be orchestrated to perform tasks using any of the previously discussed patterns. Sub-agents could be configured to work in parallel, in sequence, etc. This allows for:</p><ul><li>Extreme flexibility and mutability of the overall pattern</li><li>Complex task decompisition and execution</li><li>Hiearchcial state management for effective cross-agent data exchange and collaboration</li></ul><p>At the same time, the hierachical elements also introduce the risk of:</p><ul><li>Higher latency and cost, as main-agent orchestration and sub-agent spin-up/down require additional resources and tokens</li><li>Inflated agent-management overhead, as more main and sub-agents can require specialized system prompts, tool definitions, access controls, etc.</li></ul><h3>Extra: Sub-Agents as Tools</h3><p>Another powerful and common pattern, Sub-Agents as Tools, provides the primary agent with maximum control over the end-to-end workflow. It mirrors the Coordinator pattern with one key distinction: instead of delegating broad, autonomous tasks, the main agent spins up sub-agents for narrow, incremental actions-such as executing a specific tool and returning its output.</p><p>In this pattern, sub-agents are stateless. They do not retain context or share information with any entity other than the main agent. This ensures the primary agent maintains absolute authority over the workflow’s state and memory, preventing “context drift” across the system.</p><p>Because this pattern centralizes control, it requires precise instruction sets. You must ensure that sub-agents provide meaningful, structured data and that the main agent is sophisticated enough to synthesize these incremental steps into a final result.</p><h3>Closing Thoughts</h3><p>Agentic design patterns are constantly evolving. In production, the right pattern is often a hybrid or adaptation of the basic patterns described in this blog.</p><p>In Figure 6, I provide a simple cheatsheet summarizing some key takeaways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zTzO0Jw-7BvoQ24I" /><figcaption>Figure 6. A table summarizing the 6 design pattern’s pros, cons, and best fit use-cases.</figcaption></figure><p>If you’re implementing a unique pattern or are aware of one, I’d love to learn about it in the comments. Thanks for reading.</p><p><em>Disclaimer: This is a personal opinion piece and in no way reflects the strategy or the position of my current employer on the topic discussed.</em></p><p><strong><em>Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6825cd989178" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolutionary Pseudocode with AI in the Loop]]></title>
            <link>https://eduand-alvarez.medium.com/evolutionary-pseudocode-with-ai-in-the-loop-bf4fca7eac97?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf4fca7eac97</guid>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[evolution]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[biology]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Wed, 13 Aug 2025 18:34:34 GMT</pubDate>
            <atom:updated>2025-09-10T16:26:40.279Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JrNGRnpuL8VobsvaX6LJ-w.png" /><figcaption>Image Property of Author — Generated using AI</figcaption></figure><h4>Exploring how AI could be the Latest Nested Loop in Evolutionary Biology</h4><p>When we consider evolution in the terms <a href="https://en.wikipedia.org/wiki/Darwinism">Darwin first introduced</a>, we think on the scale of thousands of years. It turns out evolutionary biology postulates that there are much faster adaptive evolutionary mechanisms that take place in days or weeks.</p><p>I came across this concept while listening to an <a href="https://www.youtube.com/watch?v=wMavKrA-4do">episode of the Lex Fridman podcast with Manolis Kellis</a>, Head of the Harvard Computational Biology Group. In the episode, they explored a fascinating idea around evolution where, hidden inside the slow evolutionary process, are faster loops — miniature, high-speed selection environments that run well within an organism’s lifetime.</p><p>It occurred to me that this concept not only gives us insight into how nature works, but also into how modern AI might fundamentally reshape the evolutionary process for humanity. Let’s explore this using the concept of <a href="https://www.w3schools.com/python/gloss_python_for_nested.asp">nested loops in software</a>.</p><h4>1. The Outer Loop: Biological Evolution</h4><p>The outer loop represents the long game. It occurs over many thousands of years, where changes in DNA accumulate in response to environmental pressures. For humans, this loop is slow — our generation times are long, mutation rates are low, and the most meaningful changes require many lifetimes to manifest.</p><p>If you subscribe to Darwinian evolutionary theory, this outer loop can be thought of as solely powered by natural selection:</p><ul><li>Traits that improve survival and reproduction spread, while traits that don’t fade away over generations.</li><li>Selection pressures come from the environment, predators, competition, and resource availability.</li></ul><p>This loop is a patient process that requires massive amounts of data and factors to impact change — yet the pseudocode could be quite simple:</p><pre>for generation in human_species:<br>    for individual in population:<br>        if survives_environment(individual):<br>            reproduce(individual)</pre><p>It’s gotten us this far, but it has a downside — it is extremely slow and doesn’t help humans adapt to rapidly changing threats, such as viruses and bacteria, which mutate and adapt in weeks or months.</p><h4>2. The First Inner Loop: High-Speed Selection Within a Lifetime</h4><p>This was the idea explored by Manolis on Lex’s podcast. Nature compensates for slow genetic cycles in the outer loop with fast, within-lifetime tests. These inner loops are like biological quality-control systems that can quickly filter and optimize without waiting a generation.</p><p>A fascinating example of this can be found in human reproduction — specifically, the journey of sperm to fertilize an egg. In this microscopic race, the sperm undergo:</p><ul><li><strong>Viability checks </strong>that remove defective sperm early.</li><li><strong>Mobility challenges </strong>that filter out those unable to navigate the reproductive tract.</li><li><strong>Immune defenses</strong> that destroy weak sperm.</li><li><strong>Compatibility checks </strong>that ensure only those with the correct biochemical signatures can fertilize the egg.</li></ul><p>Millions of candidates enter — only one passes all of the tests. We can represent this as a Python loop with assertions testing different qualities of the candidate. This loop provides an adaptive evolutionary micro-tournament — millions of trials in minutes, not centuries.</p><pre>for sperm in population:<br>    assert is_viable(sperm)<br>    assert navigates_environment(sperm)<br>    assert survives_immune_response(sperm)<br>    assert binds_and_fuses_with_egg(sperm)</pre><p>These microscopic competitions are just one kind of inner loop. Humans also have another — one that doesn’t depend on biology at all.</p><h4>3. The Middle Loop: Cultural Evolution</h4><p>The nested evolutionary concept got me thinking — what other loops could exist? Another one that is fairly obvious is non-biological, and that’s culture. This inner loop allows us to evolve not just through DNA, but through shared knowledge, technology, and ideas.</p><p>When a new human is born, they come with specific genetic traits out of the box (OOTB). However, most of our human experience is learned — this is the common <a href="https://en.wikipedia.org/wiki/Nature_versus_nurture">nature vs. nurture debate</a>. If we subscribe to a predominantly nurture-based existence, then that means every new human experiences many unique downloads of the state of humanity throughout their lifetime.</p><p>Let’s think of it as versions of software being installed on progressively more capable hardware. In the first few years of life, we are fairly simple compute systems with 2GB of RAM, no dedicated GPU, and a basic CPU — this limits the kind of software that can be installed at that time, and therefore limits the version of the human experience that can be captured. As we age into the prime of our life, the capabilities of our hardware systems grow, and we become capable of installing advanced versions of the human experience.</p><p>That’s where cultural evolution has its greatest impact and can lead to faster iteration and adaptation than biological evolution:</p><ul><li>We can learn new farming techniques in a season.</li><li>We can create new words and social norms to help us navigate life.</li><li>We can create and spread technologies in less than a year.</li></ul><p>Each major step in cultural evolution updates the version of the human experience we download and install. It can come with helpful patches, but also with bugs, viruses, and regressions. This is where this evolutionary loop evaluates whether the cultural evolution step helps us move the needle in the right direction. While slower than biology’s first inner loop, cultural evolution helps us adapt through the human experience in ways genes cannot.</p><pre># What each person installs and keeps over time<br>for age in life_stages:<br>    capacity = cognitive_bandwidth(age)           # time, attention, plasticity<br>    for idea in available_ideas(age):<br>        if teachability(idea) &lt;= capacity and trust(idea) &gt;= min_trust(age):<br>            install(person, idea)<br><br>    # keep what proves useful; forget what doesn’t<br>    for idea in list(installed(person)):<br>        if real_world_payoff(person, idea) &lt; keep_threshold(person):<br>            uninstall(person, idea)</pre><p>Now, another loop is emerging, one that could make even cultural evolution seem slow.</p><h4>4. Inner Loop 2.0: AI as a Hyper-Accelerated Evolutionary Testbed</h4><p>In 2025, we stand on the edge of a new kind of loop, one created by the growing capabilities and large-scale adoption of artificial intelligence. Elements of cultural evolution can now be tested not in years or decades, but in minutes and seconds.</p><p>Here’s why AI is different:</p><ul><li><strong>Speed:</strong> Billions of simulations, optimizations, or experiments can run in parallel.</li><li><strong>Breadth:</strong> AI can explore a wider range of possibilities than a human mind could consider.</li><li><strong>Stacked assert functions:</strong> Outputs can be automatically filtered for performance, ethics, cost, and feasibility before humans even see them.</li></ul><p>The impact of this concept is profound, let’s look at it through the lens of the hardware and software analogy I used in the previous section. Imagine that instead of having minor releases of software to download and install across your lifetime, the software now jumps entire major versions at the rate you previously only had to download and install minor ones — or even crazier, the entire programming paradigm is redefined every year, like a user having to relearn their entire UX with each weekly patch.</p><p>This means your hardware’s instruction sets, memory speeds, thread counts, and thermal properties might need to be overhauled at a rapid pace just to download the latest version of the human experience. This could cause tremendous stress on the system, so we would need further adaptations to handle this rapid change, or lean on systems that can help us carry the load.</p><pre>RELEASES_PER_DAY = 3              # AI generates &amp; ships continuously<br>capacity_per_day = 1              # human attention/time per day<br>installed = 0<br>backlog   = 0<br><br>for day in lifespan_days:<br>    updates = RELEASES_PER_DAY<br><br>    # naive attempt to install everything, like constant “major jumps”<br>    install_cost = updates<br>    can_install  = min(install_cost, capacity_per_day)<br><br>    installed += can_install<br>    backlog   += (install_cost - can_install)    # queue grows fast</pre><p>And just like biology uses selection to ensure quality, we may need AI systems to filter, compress, and batch updates before they reach us.</p><h4>5. Digital Evolution Core Loop: AI High-Speed Selection Within Days</h4><p>This is where it could get even more interesting — the AI loop’s acceleration of cultural evolution could potentially lead to the need for an AI-Human compatible evolutionary system. This could be similar to the human reproductive process, where many AI systems attempt to find a fit with a human running a particular “software version” of the human experience. The AI system would be put through a series of compatibility tests to determine if they ultimately, together with the human, yield an augmentation that helps them navigate and succeed in the hyper-accelerated cultural evolutionary testbed.</p><pre>for ai in ai_candidates:<br>    assert is_reliable(ai)                    <br>    assert fits_user_environment(ai)     <br>    assert passes_safety_and_oversight(ai)<br>    assert aligns_and_improves_outcomes(ai)</pre><p>One example of AI’s role in this mechanism is to provide coping strategies. Again using software as an example, AI can serve as a compression engine — providing filters and pre-download/installation tests that help us reduce our cultural evolutionary backlog and potentially skip versions that are either not useful or potentially harmful to our particular human experience.</p><pre>RELEASES_PER_DAY     = 3<br>capacity_per_day     = 1<br>compression_factor   = 0.5   # AI summarizes/auto-migrates (cuts install cost)<br>batch_interval_days  = 7     # weekly rollups instead of drip-drip<br>policy_gate          = lambda u: u.is_safe and u.is_useful<br><br>installed = 0<br>backlog   = []<br><br>for day in lifespan_days:<br>    # filter + compress before human installs<br>    todays = [u for u in generate_updates(n=RELEASES_PER_DAY) if policy_gate(u)]<br>    todays_cost = len(todays) * compression_factor<br><br>    backlog.append((todays, todays_cost))<br><br>    # batch-install on schedule to reduce context switching<br>    if day % batch_interval_days == 0:<br>        batch_updates, batch_cost = merge(backlog)    # rollup + dedupe<br>        can_install = min(batch_cost, capacity_per_day * batch_interval_days)<br>        installed  += can_install<br>        backlog     = [] if can_install == batch_cost else residual(batch_updates, batch_cost, can_install)</pre><p>The animation below (Figure 1) illustrates a potential workflow in which candidate AI systems are evaluated for compatibility with both the human user and the latest batch of high-speed cultural updates. Following this evaluation, selected systems act as adaptive filters, performing compression, synthesis, and prioritization, to reduce cognitive load. This enables the human to more efficiently assimilate the most relevant and beneficial elements of the evolving “human experience” while discarding low-value or harmful updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*3c6LFKCsOt2x8_F172uCkA.gif" /><figcaption>Figure 1. Animation showing AI systems filtered for human compatibility to compress and prioritize rapid cultural updates.</figcaption></figure><h4>Closing Thougths and Disclaimers</h4><p>In case it isn’t obvious, I’m not an evolutionary biologist, but I find it fascinating to explore where different disciplines intersect with AI and software. This piece started as a spark from the conversation between Lex and Manolis and quickly took on a life of its own as I began to hypothesize about how AI could influence human evolution, and how deeply our digital lives are now intertwined with our biology.</p><p>It seems entirely plausible to me that, over the long term, the stress of rapid AI-driven iteration on the “human operating system” could only be balanced, and fully assimilated, by Human-AI collaborations. Whether this evolutionary biology take on AI is an idea already explored, or complete hogwash, I can’t say for sure.</p><p>What I do know is that it’s fun and thought-provoking to imagine these possibilities. This is simply my take on the matter, and I’d love to hear yours — especially if you know of any active research exploring this space.</p><p><em>Disclaimer: This is a personal opinion piece and in no way reflects the strategy or the position of my current employer on the topic discussed.</em></p><p><strong><em>Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf4fca7eac97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are LLM Tokens Worth?]]></title>
            <link>https://eduand-alvarez.medium.com/what-are-llm-tokens-worth-ac325315f8ea?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac325315f8ea</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[tokenomics]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Sat, 03 May 2025 15:00:03 GMT</pubDate>
            <atom:updated>2025-05-04T22:28:05.517Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebJZoOi49jYwF3A2TZQFFQ.png" /><figcaption>Image is Property of the Author</figcaption></figure><h4>An initial dive into how LLM tokens translate into the economic impact for humans and business</h4><p><strong>LLMs aren’t just generating text — they’re generating economic output, one token at a time. </strong>To understand their actual impact, we need to look beyond performance benchmarks and explore what those tokens are worth in the real world.</p><p>We can measure the output of large language models (LLMs) in terms of throughput — number of tokens generated per unit of time. In traditional performance benchmarking, this is where we stop — with raw performance. However, to truly understand the real-world value of language models, we have to go a step further and analyze the economic impact of these assets on the businesses and end-users deploying and consuming AI services. This article unpacks the economics of LLM tokens [1].</p><h3>The Mechanics of Token-Based Pricing</h3><p>LLMs break down text into tokens, where a token might be a word fragment or punctuation. Current API pricing models for premium models such as GPT-4 are typically around <a href="https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost"><strong>$0.03–$0.06 per 1,000 tokens</strong></a>. This model allows for a near-microtransaction approach to pricing — every generated token is a quantifiable unit of cost.</p><p>Some companies might seek to directly integrate an API into a specific component of their application — for example, building an AI customer service agent that pipes all queries to the OpenAI API. Others will seek to have bespoke deployments of AI models in their own data centers or leverage the hardware from a cloud service provider (CSP). The varying models for deploying these solutions can lead to significant differences in the price they assign to the tokens the business serves.</p><p>The price a business sets for a token ultimately boils down to the equation in Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h5zeFHGivhvP5oKPMZ6-cQ.png" /><figcaption>.Figure 1: Token cost equation graphic. This equation was developed by the author as a common sense way to approximate what an AI service provider might charge per token given the technical/business overhead and any premiums they define for their service. — Image by author</figcaption></figure><p><strong>Let’s break this down:</strong></p><ul><li><strong>Cost to Generate the Token:</strong> This includes everything required to serve the token — hardware, infrastructure, engineering, software licensing, and general operational overhead.</li><li><strong>Premium:</strong> The premium reflects the differentiated value the token provides. A token generated by a closed-source, proprietary model solving a high-value use case can command a much higher premium than one produced by a basic open-source model behind a simple API endpoint. This premium can also account for the service’s reliability and user/data safety.</li></ul><h3>Justifying the Cost of a Token</h3><p>Regardless of the use case, the cost per token has to be justified by <em>something</em>. For simple API-based models, this is relatively straightforward. However, things get trickier when you try to assess the value of a token in more complex applications or business models.</p><p>Take Meta, for example (Figure 2). What’s a single token worth in the context of Meta AI integrated into Instagram? Since it’s offered as a freemium service, it’s likely a <a href="https://en.wikipedia.org/wiki/Loss_leader">loss leader </a>— something that brings intrinsic value tied to Instagram’s core revenue engine: ad revenue. If Meta AI makes Instagram a more engaging platform, it could drive more users to spend more time there, ultimately leading to greater ad consumption. In that case, the value of a token might be understood as a function of increased user engagement and the resulting ad impressions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4A7bzb3z8vw9bVKr" /><figcaption>Figure 2: Diagram of Meta AI token value. This concept was developed by the author in an effort to provide an example of how token value could be perceived by a business like Meta. However, it is not based on any statements made by Meta about the economics of their Meta AI platform. — Image by Author</figcaption></figure><p>Alternatively, if Meta AI replaces other methods Meta previously used to attract users — like manual content curation, influencer partnerships, or promotional campaigns — then each token could represent cost savings. In this view, tokens are valuable not just because they drive revenue, but because they reduce the need for other expensive acquisition tactics.</p><p>Either way, the value of a token can be seen as either revenue gained through augmentation, costs saved through replacement, or both. There are other AI monetization models like subscription-based, SaaS Bundling, Outcome-based, and freemium, but those are out of the scope of this article.</p><h3>Role of Token Economics in Human Worker Augmentation</h3><p>The following section calculates the cost per token for a popular model like Llama 3.1 70B Instruct. First, we consider the average throughput of common AI accelerators — <a href="https://www.amd.com/en/developer/resources/rocm-hub/dev-ai/performance-results.html">MI300X</a> and <a href="https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/performance/perf-overview.md">H100</a> processors — as reported on their respective public pages. Then, we use the public hourly pricing from Oracle Compute Cloud to determine the cost per token.</p><p>At the time of writing, using a configuration with <strong>2048 input tokens and 2048 output tokens</strong>, the GPUs achieved an average output token throughput of<strong> 8,500 tokens/sec</strong> across both processors using 8-bit quantized versions of the 70B model deployed using tensor parallelism across the 8 GPUs on the node. The average cost of an 8‑GPU node (average of H100 and MI300X on-demand cost) on <a href="https://www.oracle.com/cloud/price-list/">Oracle Cloud</a> is approximately <strong>$64 per hour</strong>. Converting this throughput to an hourly rate gives about 30 million tokens per hour (8,500 tokens/s × 3600 s). We divide the hourly cost by the number of tokens produced and we get an approximate cost of <strong>$.000002 per token </strong>or <strong>$2.09 per million output tokens</strong>.</p><p>We will add a 20% buffer for maintenance/troubleshooting-related downtime for the analysis below — bringing the<strong> tokens per hour down to 24 million</strong>. I’ll also factor in the cost of a small engineering team (3 engineers @ $100/hour each), which will be the dominant cost in the overall token economics. This brings the total cost per token to<strong> $0.000014 </strong>or <strong>$14.87 per million output tokens.</strong></p><p><em>Note: These values can change significantly with new optimizations, pricing adjustments, model sizes, and architecture variations.</em></p><h4>Looking at Real World Use-Cases</h4><p>Across various industries, AI-generated tokens translate to dramatic value when used to augment or partially replace human workers. Using the token economics of the analysis above, we apply them to three common use cases in healthcare, finance, and customer service. Table 1 provides a technical breakdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UzR4Qr__-TOsZ63BgLnqww.png" /><figcaption>Table 1: Detailed analysis of AI cost and human cost for common tasks across healthcare, finance, and customer service. The key figures of merit are the implied value of tokens through savings, which helps address the core topic of this article: “What are tokens worth” — Image by author</figcaption></figure><p>To perform the evaluation that resulted in the metrics in the table above, we made a few key assumptions and mined data from public sources:</p><ol><li><strong>Estimate Task Size (in Words): </strong>For each use case, we researched typical examples to determine how many words a human would produce or process to complete the task.</li><li><strong>Convert Words to Tokens: </strong>Using the standard English conversion rate of ~0.75 words per token, we translated each task into a token count suitable for LLM completion.</li><li><strong>Determine AI Token Cost: </strong>Based on earlier analysis — including cloud compute costs and engineering overhead — we calculated the cost of generating those tokens.</li><li><strong>Estimate Human Labor Cost: </strong>We used average hourly wages (sourced from the U.S. Bureau of Labor Statistics) for relevant professions to estimate the human cost per task.</li><li><strong>Scale to 100 Tasks: </strong>To demonstrate impact at scale, we projected costs for 100 instances of each task.</li></ol><p>From these inputs, we calculated <strong>potential savings</strong> by subtracting the total AI cost from the total human labor cost. The result gives a clear picture of how LLMs can reduce operational expenses for repetitive or routine workflows.</p><h4>Implied Token Value (ITV)</h4><p>A central metric in evaluating the economic impact of AI tokens is the <strong>Implied Value of Tokens Through Savings</strong>. This figure encapsulates the core inquiry: <em>“What is the token worth?”. </em>It is calculated using the equation in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*b0HZAHg-ReRsCYevEJQJCA.png" /><figcaption>Figure 3: Equation for calculating the implied token value through savings. This equation was developed by the author as a method for calculating individual token values for specific tasks. It does not consider factors like degradation in token value caused by hallucinations or erroneous outputs from LLMs. — Image by Author</figcaption></figure><p>In this context, the token’s value is inherently tied to its capacity to reduce costs by augmenting or replacing human labor.</p><h4>Example of Financial Report Analysis</h4><p>Consider the scenario of financial report analysis (Figure 4). Here, each token contributes an estimated <strong>$0.09</strong> in net value to the business after accounting for associated costs. This value arises from the token’s ability to either replace or enhance human efforts in this task. Assuming the cost per token equates to the expense of employing an AI worker — approximately <strong>$0.000014</strong> — we derive a <strong>wage-to-benefit ratio of 6,700x</strong>. This ratio starkly contrasts with traditional human labor metrics, where businesses typically aim for a <a href="https://smallbusinessconnections.com.au/what-percent-of-your-revenue-should-go-to-profits-wages-and-rent-the-breakdown-every-smb-needs/?utm_source=chatgpt.com">revenue-to-wage ratio between <strong>2.5x and 3x</strong></a>. Such human labor ratios are designed to cover wages, overhead, and profit margins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v7K0jL7vqpHyEcFcOZmsiQ.png" /><figcaption>Figure 4: Financial report analysis example graphic. This figure further helps illustrate part of the analysis in Table 1. Note again, that these examples are approximations and don’t consider all possible factors involved in delivering a final product for this task. — Image by Author</figcaption></figure><p>While it’s essential to acknowledge potential caveats — such as scalability challenges and the need for additional resources to serve a broader user base — the substantial <strong>6,700x ratio</strong> offers considerable leeway. It provides ample room to accommodate increased costs while still maintaining significant benefits from integrating AI into this use case.</p><p>This analysis highlights a significant efficiency opportunity — <strong>not to replace humans but to offload routine work, freeing people for higher-value tasks.</strong></p><blockquote>Note: The token generation costs used in this analysis are based solely on cloud infrastructure compute costs and a small engineering team. Additional factors — such as power, maintenance, business development, marketing, or custom deployments — may affect the total cost in other settings.</blockquote><h3>Conclusion</h3><p>As AI adoption accelerates, businesses must evolve how they think about cost, performance, and value. Large Language Model (LLM) tokens offer a unique unit of measure — quantifiable, monetizable, and comparable across a range of tasks.</p><p>From reducing the time it takes to triage patients or summarize financial reports, to handling customer support inquiries at scale, tokens represent potential — potential to cut costs, scale operations, and shift human labor to higher-value tasks.</p><p>While real-world deployment requires accounting for infrastructure, personnel, and variability in demand, the core insight remains powerful: <strong>each token generated by an LLM carries tangible business value. </strong>And as models get cheaper, faster, and more specialized, that value will only increase.</p><p><strong>End notes:</strong></p><p><em>[1] Please note that this article does not advocate for AI tools to completely replace human labor — which could have unforeseen, profound, and far-reaching impacts on the economic stability of households and society’s mental health.</em></p><p><strong><em>Thank you for reading! Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><p><strong><em>Disclaimer:</em></strong><em> The views expressed in this article are my own and do not reflect those of my employer. References are available upon request.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac325315f8ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meta Llama 3 Optimized CPU Inference with Hugging Face and PyTorch]]></title>
            <link>https://medium.com/data-science/meta-llama-3-optimized-cpu-inference-with-hugging-face-and-pytorch-9dde2926be5c?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/9dde2926be5c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llama-3]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Fri, 19 Apr 2024 15:36:20 GMT</pubDate>
            <atom:updated>2024-04-28T19:04:04.087Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mngqkmC_kUdvCXZ9nIKlA.jpeg" /><figcaption>Created with Nightcafe — Image property of Author</figcaption></figure><h4>Learn how to reduce model latency when deploying Meta* Llama 3 on CPUs</h4><p>The much-anticipated <a href="https://huggingface.co/blog/llama3">release</a> of Meta’s third-generation batch of Llama is here, and I want to ensure you know how to deploy this state-of-the-art (SoTA) LLM optimally. In this tutorial, we will focus on performing weight-only-quantization (WOQ) to compress the 8B parameter model and improve inference latency, but first, let’s discuss Meta Llama 3.</p><h3>Llama 3</h3><p>To date, the Llama 3 family includes models ranging from 8B to 70B parameters, with more versions coming in the future. The models come with a permissive Meta Llama 3 <a href="https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/LICENSE">license</a>, you are encouraged to review before accepting the terms required to use them. This marks an exciting chapter for the Llama model family and open-source AI.</p><h4>Architecture</h4><p>The Llama 3 is an auto-regressive LLM based on a decoder-only transformer. Compared to Llama 2, the Meta team has made the following notable improvements:</p><ul><li>Adoption of grouped query attention (GQA), which improves inference efficiency.</li><li>Optimized tokenizer with a vocabulary of 128K tokens designed to encode language more efficiently.</li><li>Trained on a 15 trillion token dataset, this is 7x larger than Llama 2’s training dataset and includes 4x more code.</li></ul><p>The figure below (Figure 1) is the result of print(model) where model is <a href="https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct">meta-llama/Meta-Llama-3–8B-Instruct</a>. In this figure, we can see that the model comprises 32 LlamaDecoderLayers composed of Llama Attention self-attention components. Additionally, it has LlamaMLP, LlamaRMSNorm, and a Linear head. We hope to learn more once the Llama 3 research paper is released.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NqxEdTub3iJTw4ipT6Jg8Q.png" /><figcaption>Figure 1. Output of `print(model)` showcasing the distribution of layers across llama-3–8B-instruct’s architecture — Image by Author</figcaption></figure><h4>Language Modeling Performance</h4><p>The model was evaluated on various industry-standard language modeling benchmarks, such as MMLU, GPQA, HumanEval, GSM-8K, MATH, and more. For the purpose of this tutorial, we will review the performance of the “Instruction Tuned Models” (Figure 2). The most remarkable aspect of these figures is that the Llama 3 8B parameter model outperforms Llama 2 70B by 62% to 143% across the reported benchmarks while being an 88% smaller model!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OL1IH76N9yOpoeRLv2BTwA.png" /><figcaption>Figure 2 . Summary of Llama 3 instruction model performance metrics across the MMLU, GPQA, HumanEval, GSM-8K, and MATH LLM benchmarks. — Image by Author (<a href="https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct">source</a>)</figcaption></figure><p>The increased language modeling performance, permissive licensing, and architectural efficiencies included with this latest Llama generation mark the beginning of a very exciting chapter in the generative AI space. Let’s explore how we can optimize inference on CPUs for scalable, low-latency deployments of Llama 3.</p><h3>Optimizing Llama 3 Inference with PyTorch</h3><p>In a <a href="https://medium.com/towards-data-science/improving-llm-inference-latency-on-cpus-with-model-quantization-28aefb495657">previous article</a>, I covered the importance of model compression and overall inference optimization in developing LLM-based applications. In this tutorial, we will focus on applying weight-only quantization (WOQ) to <a href="https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct">meta-llama/Meta-Llama-3–8B-Instruct</a>. WOQ offers a balance between performance, latency, and accuracy, with options to quantize to int4 or int8. A key component of WOQ is the dequantization step, which converts int4/in8 weights back to bf16 before computation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*RH6ou7jL5Fw9KwGG.png" /><figcaption>Fig 3. Simple illustration of weight-only quantization, with pre-quantized weights in orange and the quantized weights in green. Note that this depicts the initial quantization to int4/int8 and dequantization to fp16/bf16 for the computation step. — Image by Author (<a href="https://medium.com/towards-data-science/improving-llm-inference-latency-on-cpus-with-model-quantization-28aefb495657">source</a>)</figcaption></figure><h4>Environment Setup</h4><p>You will need approximately 60GB of RAM to perform WOQ on Llama-3-8B-Instruct. This includes ~30GB to load the full model and ~30GB for peak memory during quantization. The WOQ Llama 3 will only consume ~10GB of RAM, meaning we can free ~50GB of RAM by releasing the full model from memory.</p><blockquote><em>You can run this tutorial on the </em><a href="https://medium.com/r?url=https%3A%2F%2Fbit.ly%2Fllama3woq"><em>Intel</em>®<em> Tiber</em>®<em> Developer Cloud</em></a><em> free JupyterLab* environment. This environment offers a 4th Generation Intel</em>®<em> Xeon</em>®<em> CPU with 224 threads and 504 GB of memory, more than enough to run this code.</em></blockquote><p>If running this in your own IDE, you may need to address additional dependencies like installing Jupyter and/or configuring a conda/python environment. Before getting started, ensure that you have the following dependencies installed.</p><pre>intel-extension-for-pytorch==2.2<br>transformers==4.35.2<br>torch==2.2.0<br>huggingface_hub</pre><h4>Accessing and Configuring Llama 3</h4><p>You will need a Hugging Face* account to access Llama 3’s model and tokenizer.</p><p>To do so, select “Access Tokens” from your settings menu (Figure 4) and create a token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*nC0ovX4kgCkxqQTZeeHXXA.png" /><figcaption>Figure 4. Snapshot of the Hugging Face token configuration console — Image by Author</figcaption></figure><p>Copy your access token and paste it into the “Token” field generated inside your Jupyter cell after running the following code.</p><pre>from huggingface_hub import notebook_login, Repository<br><br># Login to Hugging Face<br>notebook_login()</pre><p>Go to <a href="https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct">meta-llama/Meta-Llama-3–8B-Instruct</a> and carefully evaluate the terms and license before providing your information and submitting the Llama 3 access request. Accepting the model’s terms and providing your information is yours and yours alone.</p><h4>Quantizing Llama-3–8B-Instruct with WOQ</h4><p>We will leverage the <a href="https://medium.com/r?url=https%3A%2F%2Fwww.intel.com%2Fcontent%2Fwww%2Fus%2Fen%2Fdeveloper%2Ftools%2Foneapi%2Foptimization-for-pytorch.html">Intel® Extension for PyTorch</a>* to apply WOQ to Llama 3. This extension contains the latest PyTorch optimizations for Intel hardware. Follow these steps to quantize and perform inference with an optimized Llama 3 model:</p><ol><li><strong>Llama 3 Model and Tokenizer:</strong> Import the required packages and use the AutoModelForCausalLM.from_pretrained() and AutoTokenizer.from_pretrained() methods to load the Llama-3–8B-Instruct specific weights and tokenizer.</li></ol><pre>import torch<br>import intel_extension_for_pytorch as ipex<br>from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer<br><br>Model = &#39;meta-llama/Meta-Llama-3-8B-Instruct&#39;<br><br>model = AutoModelForCausalLM.from_pretrained(Model)<br>tokenizer = AutoTokenizer.from_pretrained(Model)</pre><p>2. <strong>Quantization Recipe Config:</strong> Configure the WOQ quantization recipe. We can set the weight_dtype variable to the desired in-memory datatypes, choosing from torch.quint4x2 or torch.qint8 for int4 and in8, respectively. Additionally we can use lowp_model to define the dequantization precision. For now, we will keep this as ipex.quantization.WoqLowpMode.None to keep the default bf16 computation precision.</p><pre>qconfig = ipex.quantization.get_weight_only_quant_qconfig_mapping(<br>  weight_dtype=torch.quint4x2, # or torch.qint8<br>  lowp_mode=ipex.quantization.WoqLowpMode.NONE, # or FP16, BF16, INT8<br>)<br>checkpoint = None # optionally load int4 or int8 checkpoint<br><br># PART 3: Model optimization and quantization<br>model_ipex = ipex.llm.optimize(model, quantization_config=qconfig, low_precision_checkpoint=checkpoint)<br><br>del model </pre><p>We use ipex.llm.optimize() to apply WOQ and then del model to delete the full model from memory and free ~30GB of RAM.</p><p>3. <strong>Prompting Llama 3: </strong>Llama 3, like LLama 2, has a pre-defined prompting template for its instruction-tuned models. Using this template, developers can define specific model behavior instructions and provide user prompts and conversation history.</p><pre>system= &quot;&quot;&quot;\n\n You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. If you don&#39;t know the answer to a question, please don&#39;t share false information.&quot;&quot;&quot;<br>user= &quot;\n\n You are an expert in astronomy. Can you tell me 5 fun facts about the universe?&quot;<br>model_answer_1 = &#39;None&#39;<br><br>llama_prompt_tempate = f&quot;&quot;&quot;<br>&lt;|begin_of_text|&gt;\n&lt;|start_header_id|&gt;system&lt;|end_header_id|&gt;{system}<br>&lt;|eot_id|&gt;\n&lt;|start_header_id|&gt;user&lt;|end_header_id|&gt;{user}<br>&lt;|eot_id|&gt;\n&lt;|start_header_id|&gt;assistant&lt;|end_header_id|&gt;{model_answer_1}&lt;|eot_id|&gt;<br>&quot;&quot;&quot;<br><br>inputs = tokenizer(llama_prompt_tempate, return_tensors=&quot;pt&quot;).input_ids</pre><p>We provide the required fields and then use the tokenizer to convert the entire template into tokens for the model.</p><p>4.<strong> Llama 3 Inference:</strong> For text generation, we leverage TextStreamer to generate a real-time inference stream instead of printing the entire output at once. This results in a more natural text generation experience for readers. We provide the configured streamer to model_ipex.generate() and other text-generation parameters.</p><pre>streamer = TextStreamer(tokenizer,skip_prompt=True)<br><br>with torch.inference_mode():<br>    tokens = model_ipex.generate(<br>        inputs,<br>        streamer=streamer,<br>        pad_token_id=128001,<br>        eos_token_id=128001,<br>        max_new_tokens=300,<br>        repetition_penalty=1.5,<br>)</pre><p>Upon running this code, the model will start generating outputs. Keep in mind that these are unfiltered and non-guarded outputs. For real-world use cases, you will need to make additional post-processing considerations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*_DRMPKM_99Z6281hSiuYHQ.gif" /><figcaption>Figure 5. Streamed inference of Llama-3–8B-Instruct with WOQ mode compression at int4 running on the Intel Tiber Developer Cloud’s JupyterLab environment — Gif by Author</figcaption></figure><p>That’s it. With less than 20 lines of code, you now have a low-latency CPU optimized version of the latest SoTA LLM in the ecosystem.</p><h4>Considerations for Deployment</h4><p>Depending on your inference service deployment strategy, there are a few things that you will want to consider:</p><ul><li>If deploying instances of Llama 3 in containers, WOQ will offer a smaller memory footprint and allow you to serve multiple inference services of the model on a single hardware node.</li><li>When deploying multiple inference services, you should optimize the threads and memory reserved for each service instance. Leave enough additional memory (~4 GB) and threads (~4 threads) to handle background processes.</li><li>Consider saving the WOQ version of the model and storing it in a model registry to eliminate the need to re-quantize the model per instance deployment.</li></ul><h3>Conclusion and Discussion</h3><p>Meta’s Llama 3 LLM family delivers remarkable improvements over previous generations with a diverse range of configurations (more coming soon). In this tutorial, we explored enhancing CPU inference with weight-only quantization (WOQ), a technique that reduces latency with minimal impacts to accuracy.</p><p>By integrating the new generation of performance-oriented Llama 3 LLMs with optimization techniques like WOQ, developers can unlock new possibilities for <a href="https://www.intel.com/content/www/us/en/developer/topic-technology/artificial-intelligence/training/generative-ai.html">GenAI</a> applications. This combination simplifies the hardware requirements to achieve high-fidelity, low-latency results from LLMs integrated into new and existing systems.</p><p><strong>A few exciting things to try next would be:</strong></p><ol><li><strong>Experiment with Quantization Levels:</strong> You should test int4 and int8 quantization to identify the best compromise between performance and accuracy for your specific applications.</li><li><strong>Performance Monitoring:</strong> It is crucial to continuously assess the performance and accuracy of the Llama 3 model across different real-world scenarios to ensure that quantization maintains the desired effectiveness.</li><li><strong>Test more Llamas: </strong>Explore the entire Llama 3 family and evaluate the impact of WOQ and other PyTorch <a href="https://pytorch.org/docs/stable/quantization.html">quantization recipes</a>.</li></ol><p><strong><em>Thank you for reading! Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><p><em>*Other names and brands may be claimed as the property of others.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dde2926be5c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/meta-llama-3-optimized-cpu-inference-with-hugging-face-and-pytorch-9dde2926be5c">Meta Llama 3 Optimized CPU Inference with Hugging Face and PyTorch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transforming Financial Services with RAG: Personalized Financial Advice]]></title>
            <link>https://eduand-alvarez.medium.com/transforming-financial-services-with-rag-personalized-financial-advice-f58de9d2d7e0?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/f58de9d2d7e0</guid>
            <category><![CDATA[financial-services]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Fri, 05 Apr 2024 20:56:39 GMT</pubDate>
            <atom:updated>2024-04-11T16:58:28.076Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqKvF90_6Ynh0uhy1gjmyA.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><h4>Synthesize the complex web of financial strategies, regulations, and trends with a personalized financial advice RAG-based chatbot</h4><p>The financial services industry faces a challenge in balancing personalization and scale, as customer expectations for timely, relevant, and customized financial advice continue to rise. Traditional solutions may struggle to meet these demands due to the complexity and rapid evolution of financial data and regulations. A Retrieval-Augmented Generation (RAG) solution can significantly enhance financial advice by providing real-time access to up-to-date regulatory, market, and financial product data and synthesizing it into actionable advice for users.</p><p>However, due to the low fault-tolerance in finance, the use of non-deterministic methods such as neural-networks must be carefully evaluated. Solution developers should consider fine-tuning in addition to RAG to increase alignment and reduce risk of hallucinations.</p><h4>When are are fine-tuning and RAG appropriate?</h4><p>RAG applications generally aim to reduce the need for fine-tuning base models, but certain high-stakes sectors, like financial services and healthcare, might require extra model alignment with fine-tuning on a comprehensive knowledge-base. The necessity of both fine-tuning and RAG depends on the specific use case, available resources, necessary customer knowledge, and desired custom behavior. The appropriate balance ensures accurate, reliable, and tailored AI-driven advice.</p><p>The figure below (Figure 1) provides a simplified decision-making framework to help assess these factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k0sECDz1wYFX9rCmPp_yzQ.png" /><figcaption>Figure 1. The diagram provides a simple framework for defining when (1) RAG and Pre-trained LLM are sufficient (2) RAG and fine-tuning are required (3) A pre-trained LLM without RAG will do the job or (4) a model must be trained from scratch or fine-tuned from a foundational model and used without RAG. — Image by Author</figcaption></figure><h3>RAG-Powered Financial Advice Chatbots</h3><p>Financial institutions face the challenge of delivering high-quality, personalized financial advice to a large customer base while dealing with the continuous flux of financial data and an intricate regulatory landscape. This complex environment requires solutions that meet the growing demands and expectations of customers.</p><p>A RAG-powered financial advice chatbot can help tackle these challenges by leveraging real-time access to current financial data and regulations. Below, we present a solution architecture and a high-level overview of this approach (Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6SVTVO2YGbYWwrUmSQOoJw.png" /><figcaption>Figure 2. <strong>RAG-based Financial Advice Chatbot System Architecture. </strong>In addressing the low fault tolerance of the financial application, a foundational model is fine-tuned using relevant financial data and stored in a model registry for RAG pipeline use. The knowledge base for fine-tuning and RAG consists of data from 3rd party APIs and business-specific elements, processed and embedded for vector database construction. Upon user-query or system automation, the relevant context is retrieved, re-ranked, and personalized, then passed to the LLM inference service, utilizing the fine-tuned finance LLM. The guardrailed LLM output is then used in a financial figure/report generation process, with the final output delivered to the user through the interface, ensuring precise, context-aware, and secure financial advice.— Image by Author</figcaption></figure><p><strong>Building a Custom Knowledge Base: </strong>At the core of this RAG-enhanced chatbot is a custom knowledge base that integrates third-party and proprietary business data (Figure 3). This knowledge base informs both the data retrieval mechanisms and the fine-tuning of the chatbot’s LLM, ensuring the delivery of advice that’s accurate, relevant, and in line with the latest financial regulations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSUUhcVgFHHfOZe6nqRyxg.png" /><figcaption>Figure 3. Highlighting the <strong>Custom Knowledge Base</strong> Layer of the solution architecture— Image by Author</figcaption></figure><p><strong>Tailoring Advice with Advanced Data Processing: </strong>When a client interacts with the chatbot or a pre-configured alert triggers a system action, a customized process is initiated, which includes data retrieval, re-ranking, and personalization (Figure 4). This tailored approach ensures that the advice provided is not generic but specifically adapted to the client’s distinct financial situation and objectives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BsNselkV38AKgmcT-uQEg.png" /><figcaption>Figure 4. Highlighting the <strong>Advanced Data Processing</strong> Layer of the solution architecture— Image by Author</figcaption></figure><p><strong>Ensuring Accuracy and Compliance: </strong>To maintain precision and adherence to regulations, the chatbot’s outputs are subjected to stringent guardrails and post-processing (Figure 5). This step is critical in the financial sector, where the cost of inaccuracies can be substantial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J9aQKJHCWhv2G2h4c0cxQw.png" /><figcaption>Figure 5. Highlighting the <strong>Accuracy and Compliance</strong> Layer of the solution architecture— Image by Author</figcaption></figure><p><strong>The Role of Visual Aids and Reports in Financial Advice: </strong>Financial advice relies on both the precision of the data and the clarity of communication. To facilitate this, the chatbot includes a feature that generates financial figures and reports, such as tables, charts, plots, account statements, and other visualizations. These tools help clients better understand and visualize their financial standing and the guidance provided, ensuring that the advice is not only accurate but also comprehensible and engaging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jy7NRHzD8gGHgVF-1aWhxQ.png" /><figcaption>Figure 6. Highlighting the <strong>Visual Aids and Reports</strong> Layer of the solution architecture— Image by Author</figcaption></figure><h3>Addressing Alex and Sam’s Financial Goals with RAG</h3><p>Let’s explore a sample scenario where a solution like this might create value for end-users:</p><p>Alex and Sam each have distinct financial challenges due to their differing income stability and goals. Alex grapples with the unpredictability of freelance photography income, and Sam seeks efficient pathways to early retirement. These goals could benefit from advice beyond the scope of a generic document. A RAG financial advice chatbot, with its ability to dynamically access and augment LLM responses with a vast array of financial data and regulations, could potentially help.</p><p>For Alex, the tool can retrieve and synthesize market trends and his sporadic earning statements to craft a flexible savings strategy, enabling his dream of global travel without hindering photography equipment investments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cfvn4l3YmdQZLMSS0DowiQ.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><p>Meanwhile, Sam could benefit from a tailored investment plan derived from the chatbot’s ability to retrieve and incorporate stock market stability and spending data. This ensures reliable recommendations for building a healthy stock portfolio and balanced budget — both aligned to his early retirement goals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eILqoxjhsDJzIVJAhSnP9Q.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><p>This approach, underpinned by RAG’s data processing and retrieval capabilities, could help navigate the complexities of their financial aspirations, and prove to play a pivotal role in delivering much needed personalized financial advice.</p><h3>Summary and Discussion</h3><p>The financial services industry is undergoing significant transformation, with the need for personalized and accurate financial advice rising. RAG-based chatbots offer a solution to tackle the challenge of providing timely, relevant, and customized financial advice. By leveraging real-time access to current financial data and regulations, these chatbots demonstrate in-depth knowledge and continuous self-improvement, ensuring precision and compliance.</p><h4>Explore the entire 3-part RAG in Industry Series</h4><ul><li><a href="https://medium.com/@eduand-alvarez/transforming-retail-with-rag-the-future-of-personalized-shopping-1ac0565d98ed">Transforming Retail with RAG: The Future of Personalized Shopping</a></li><li><a href="https://medium.com/um.com/@eduand-alvarez/transforming-manufacturing-with-rag-delivering-nextgen-equipment-maintenance-89861698edd6">Transforming Manufacturing with RAG: Delivering NextGen Equipment Maintenance</a></li><li><a href="https://eduand-alvarez.medium.com/transforming-financial-services-with-rag-personalized-financial-advice-f58de9d2d7e0">Transforming Financial Services with RAG: Personalized Financial Advice</a></li></ul><p><strong><em>Thank you for reading! Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><p>Explore all parts</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f58de9d2d7e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transforming Manufacturing with RAG: Delivering NextGen Equipment Maintenance]]></title>
            <link>https://eduand-alvarez.medium.com/transforming-manufacturing-with-rag-delivering-nextgen-equipment-maintenance-89861698edd6?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/89861698edd6</guid>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[manufacturing]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Wed, 03 Apr 2024 19:01:51 GMT</pubDate>
            <atom:updated>2024-04-11T16:37:16.576Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9yCohnuM7UOvJHZTeA6VVw.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><h4>Keep operations online with actionable, relevant, and effective maintenance strategies with retrieval augmented generation</h4><p>In manufacturing, equipment is the backbone of production, and unexpected downtime can have significant financial and productivity consequences. Traditional predictive maintenance models, while helpful, can sometimes miss the subtle signs of impending failure, especially in complex machinery.</p><p>A RAG-based (Retrieval-Augmented Generation) anomaly detection system can enhance the predictive maintenance landscape by providing a more insightful and proactive approach to maintaining equipment. By integrating real-time data and advanced retrieval techniques, RAG-based systems can detect subtle signs of anomalies and predict failures before they occur, ensuring the smooth operation of manufacturing processes.</p><h3>The Cost of Unplanned Downtime</h3><p>To develop an effective solution, it’s crucial to understand the problem. Unplanned downtime in manufacturing can have a significant impact on productivity, equipment health, energy efficiency, and operational continuity. Traditional predictive maintenance, which relies heavily on historical failure data, may not be sufficient for complex systems where anomalies can be subtle and devastating.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EvUsCdIGBPYhuHup7KT5tA.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><p>Subtle anomalies can be particularly challenging to detect, as they may not be apparent in historical data. As a result, traditional predictive maintenance models may not provide accurate predictions, leading to unplanned downtime and financial losses.</p><p>To address this challenge, a more dynamic and real-time approach to predictive maintenance is needed.</p><h3>Introducing RAG for Manufacturing</h3><p>By leveraging real-time data and advanced retrieval techniques, RAG-based anomaly detection systems can provide a more insightful and proactive approach to predictive maintenance, enabling manufacturers to detect and address anomalies before they lead to unplanned downtime. Let’s take a quick look at a hypothetical example of this technique in action:</p><p>MegaAutoParts Inc. (fictitious company) manufactures car engines, relying on a high-precision robotic arm. They leverage a RAG-based anomaly detection system that monitors the robotic arm’s performance data in real-time, retrieving relevant data associated with identified anomalies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*utVKiOTBLVMItSr9By6O3g.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><p>The anomaly detection component can detect motion irregularities in a robotic arm, including subtle changes in speed and precision. When an anomaly is detected, the RAG component retrieves relevant equipment performance data from the vector database. During the retrieval phase, the system identifies records indicating that the anomaly caused bearing failure two years ago, along with the appropriate maintenance strategy to prevent failure. The anomaly and retrieved records are then processed by the LLM, generating a maintenance report.</p><p>The system alerts the maintenance team, providing detailed insights and a recommended course of action based on past successful interventions. The team schedules a precise, preventive repair during off-hours. The robotic arm is fixed before any real issue manifests, preventing production disruption and saving MegaAutoParts Inc. from delivery delays and financial loss.</p><h3>RAG-based Anomaly Detection System</h3><p>At their core, RAG-based anomaly detection systems employ RAG to cross-reference live sensor data with an extensive database of historical operational and equipment specific data. By doing so, they can spot the earliest signs of trouble — signs that might be missed by traditional systems. Below, we present a solution architecture and high-level overview of this approach (Figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WWA69L89d0zmkk46v3d_pA.png" /><figcaption>Figure 1. <strong>RAG-based Anomaly Detection System Architecture</strong>. The RAG-based anomaly detection system creates a knowledge base from historical and equipment-specific data, embedding and storing it in a vector database. A traditional ML anomaly detection model identifies anomalies, and when detected, the system creates embeddings and searches the vector database for relevant information. The retrieved information, tied to the anomaly, is passed to an LLM inference service, generating a customized maintenance recommendation. The output is guardrailed and delivered to the maintenance management system. — Image by Author</figcaption></figure><p><strong>The Predictive Maintenance Workflow: </strong>The process begins with the anomaly detection model, which continuously monitors equipment data (Figure 2). When it senses something amiss, the system springs into action, querying the RAG vector database. This database is a treasure trove of information, including everything from historical maintenance records to manufacturer guidelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v3bhoDC6y0nPUJtgFnVaFg.png" /><figcaption>Figure 2. Highlighting the <strong>Predictive Maintenance</strong> Layer of the solution architecture — Image by Author</figcaption></figure><p><strong>Generating Actionable Strategies: </strong>What sets this system apart is its ability to not just detect issues but to prescribe solutions (Figure 3). Upon detecting an anomaly, the system retrieves related past incidents and solutions, synthesizing this information through an LLM inference service to suggest a maintenance strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRF8yNx8eeAA0j-jy8QCRw.png" /><figcaption>Figure 3. Highlighting the <strong>Generating Actionable Strategies</strong> Layer of the solution architecture — Image by Author</figcaption></figure><p><strong>Integration and Implementation: </strong>A fundamental part of the solution is to ensure that final recommendations are not generic (Figure 4). They should be carefully tailored, considering the specific equipment and its operational context, ensuring the maintenance strategies are not just theoretically sound but practically actionable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3thHbwB4qLIEHQ899kmJVA.png" /><figcaption>Figure 4. Highlighting the <strong>Integration and Implementation</strong> Layer of the solution architecture — Image by Author</figcaption></figure><h4>The Benefits Are Clear</h4><p>This RAG-based anomaly detection system stands to revolutionize the industry by minimizing downtime, reducing maintenance costs, and extending the life of manufacturing equipment (Figure 5). It’s a forward-thinking solution that doesn’t just fix problems — it prevents them, ensuring that manufacturing equipment can continue its vital role in the business with minimal interruption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bOhdkIUMSB_4Ns4HNaX31Q.png" /><figcaption>Figure 5. Highlighting the key benefits of RAG in Manufacturing— Image by Author</figcaption></figure><h3>Summary and Discussion</h3><p>Anomaly detection is a critical aspect of manufacturing processes, and the RAG-based anomaly detection system is a promising solution that offers dynamic and real-time detection of subtle signs of anomalies. This system maximizes the efficiency of manufacturing processes by minimizing downtime, reducing maintenance costs, and extending the life of manufacturing equipment. With this system in place, manufacturers can ensure the smooth operation of their processes, leading to higher productivity and profitability.</p><p>By leveraging the power of RAG, the anomaly detection system provides a vital tool for manufacturers to stay ahead of potential issues and maintain optimal performance in their operations.</p><h4>Explore the entire 3-part RAG in Industry Series</h4><ul><li><a href="https://medium.com/@eduand-alvarez/transforming-retail-with-rag-the-future-of-personalized-shopping-1ac0565d98ed">Transforming Retail with RAG: The Future of Personalized Shopping</a></li><li><a href="https://medium.com/um.com/@eduand-alvarez/transforming-manufacturing-with-rag-delivering-nextgen-equipment-maintenance-89861698edd6">Transforming Manufacturing with RAG: Delivering NextGen Equipment Maintenance</a></li><li><a href="https://eduand-alvarez.medium.com/transforming-financial-services-with-rag-personalized-financial-advice-f58de9d2d7e0">Transforming Financial Services with RAG: Personalized Financial Advice</a></li></ul><p><strong><em>Thank you for reading! Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89861698edd6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transforming Retail with RAG: The Future of Personalized Shopping]]></title>
            <link>https://eduand-alvarez.medium.com/transforming-retail-with-rag-the-future-of-personalized-shopping-1ac0565d98ed?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ac0565d98ed</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[retail]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Tue, 02 Apr 2024 14:02:48 GMT</pubDate>
            <atom:updated>2024-04-06T15:59:39.427Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJe9SPN4W2oRllbVYZ7rng.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><h4>Delivering dynamic, fresh, and timely recommendations to shoppers with retrieval augmented generation</h4><p>In today’s fast-paced retail environment, traditional recommendation systems face challenges in keeping up with rapidly changing consumer preferences. These systems, which frequently rely on static data and static algorithms, can struggle to adapt to the variability of customer desires and market trends. This shortcoming can lead to a suboptimal shopping experience and a negative impact on retailers’ financial performance through missed sales and reduced customer loyalty.</p><p>The <a href="https://towardsdatascience.com/retrieval-augmented-generation-rag-inference-engines-with-langchain-on-cpus-d5d55f398502">RAG-based (Retrieval-Augmented Generation)</a> recommendation system represents a cutting-edge solution to this challenge. By leveraging advanced retrieval techniques, RAG provides up-to-the-minute recommendations tailored to individual customers’ preferences, ensuring that suggestions are as current as the latest Twitter trend and as personal as a handwritten note.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0n1ZAM6US3QJzpDgqwc-0g.png" /><figcaption>Figure 1. RAG-based Recommendation System Architecture. Live data sources pass through a pre-processing pipeline and embedding model, with the output stored in a vector database. User queries are processed, and the retrieved context from the vector database is re-ranked and personalized, with prompt filling incorporating the retrieved context. The final output is guardrailed and post-processed — Image by Author.</figcaption></figure><p>The diagram above aims to illustrate the high-level architectural elements of a RAG-based retail recommendation system. We will dive deeper into its various components and flow in this article.</p><h3>The Challenge of Personalization in Retail</h3><p>Traditional recommendation engines have long been a staple in e-commerce and brick-and-mortar retail, helping guide customers towards their next purchase. However, their inability to quickly adapt to real-time data and take into account nuanced customer feedback has led to a decrease in the relevance of their suggestions. In an industry where personalization is highly valued, this lack of relevance is a missed opportunity.</p><h3>Introducing RAG for Retail</h3><p>The RAG-based (Retrieval-Augmented Generation) recommendation system redefines personalization by dynamically integrating vast amounts of data, from market trends to individual customer interactions, into the recommendation process. This ensures that product suggestions are relevant and timely, enhancing the shopping experience and fostering stronger customer relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VrxZiJCUZ5yhSuK576bDpA.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><p>For example, when Emma, a shopper, searches for a dress for an upcoming wedding, the RAG-based system generates an embedding from her query and interacts with the vector database to find the best matches. It then re-ranks the top search results based on Emma’s previous purchases and browsing behavior and adapts the suggestions further by considering Emma’s color preferences and past feedback on dress sizes for specific brands.</p><p>Emma is presented with a curated selection of dresses that are trend-aligned and tailored to her personal style and preferences. As she interacts with the system, the RAG-based recommendation system learns from her behavior, improving the accuracy of future recommendations. Overall, the RAG-based recommendation system provides a personalized shopping experience that is up-to-date with the latest market trends.</p><h4>How RAG Works in Retail</h4><p>The magic of the RAG system unfolds across three primary layers, each playing a pivotal role in delivering personalized product suggestions (Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3IUo812Q_CXF1Unu0gnxtg.png" /><figcaption>Figure 2. Various layers of our proposed RAG retail application — Image by Author</figcaption></figure><p><strong>Data Collection and Integration Layer: </strong>This foundational layer aggregates data from diverse sources, including e-commerce platforms, CRM systems, and social media. It ensures a holistic view of customer preferences and market trends, feeding into a centralized data lake alongside a constantly updated product catalog (Figure 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B_2WM9hoF7TWivPp4-VAlA.png" /><figcaption>Figure 3. Highlighting the <strong>Data Collection and Integration Layer</strong> of the solution architecture — Image by Author</figcaption></figure><p><strong>The RAG Layer: </strong>At the heart of the system lies the RAG Layer, where the magic happens. This layer retrieves pertinent information from a vector database, which houses a rich collection of customer preferences, market trends, and product details. It’s here that the system ensures recommendations are both relevant to the current market and personalized to the individual (Figure 4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2jv8yjf-WCjnn2UKYJJBRA.png" /><figcaption>Figure 4. Highlighting the <strong>RAG Layer</strong> of the solution architecture — Image by Author</figcaption></figure><p><strong>Recommendation Generation and Personalization: </strong>Armed with insights from the RAG layer, the system dynamically generates personalized product suggestions. These recommendations are fine-tuned based on the customer’s historical data and preferences, ensuring a tailored shopping experience (Figure 5).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5uoTMh39_0K__f2AC7B3Yw.png" /><figcaption>Figure 5. Highlighting the <strong>Recommendation Generation and Personalization</strong> of the solution architecture — Image by Author</figcaption></figure><h3>Benefits of RAG in Retail</h3><p>The implementation of a RAG-based recommendation system heralds numerous benefits for retailers. Most notably, it drives sales and fosters customer loyalty through improved satisfaction. By delivering timely and relevant product suggestions, retailers can ensure their offerings resonate with customers, encouraging repeat visits and purchases (Figure 6).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9jYYgjPKs4GQa88GWZ9SqA.png" /><figcaption>Figure 6. Highlighting the key benefits of RAG in retail— Image by Author</figcaption></figure><h3>Summary and Discussion</h3><p>As the retail industry evolves, technologies like RAG (Retrieval-Augmented Generation) are becoming increasingly important for retailers who want to stay competitive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k0aBdKci74zG2wgib59OOQ.jpeg" /><figcaption>Image Created with Nightcafe — Property of Author</figcaption></figure><p>RAG has the potential to transform retail by providing real-time, personalized recommendations that are tailored to individual customers’ preferences, market trends, product inventories, etc. By leveraging these solutions and techniques, retailers can provide a more engaging and personalized shopping experience, which can lead to increased customer loyalty and revenue.</p><h4>Explore the entire 3-part RAG in Industry Series</h4><ul><li><a href="https://medium.com/@eduand-alvarez/transforming-retail-with-rag-the-future-of-personalized-shopping-1ac0565d98ed">Transforming Retail with RAG: The Future of Personalized Shopping</a></li><li><a href="https://medium.com/um.com/@eduand-alvarez/transforming-manufacturing-with-rag-delivering-nextgen-equipment-maintenance-89861698edd6">Transforming Manufacturing with RAG: Delivering NextGen Equipment Maintenance</a></li><li><a href="https://eduand-alvarez.medium.com/transforming-financial-services-with-rag-personalized-financial-advice-f58de9d2d7e0">Transforming Financial Services with RAG: Personalized Financial Advice</a></li></ul><p><strong><em>Thank you for reading! Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ac0565d98ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving LLM Inference Latency on CPUs with Model Quantization]]></title>
            <link>https://medium.com/data-science/improving-llm-inference-latency-on-cpus-with-model-quantization-28aefb495657?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/28aefb495657</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Thu, 29 Feb 2024 17:48:31 GMT</pubDate>
            <atom:updated>2024-04-01T19:50:19.687Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iS3mFlEoUWT3UT6E0RzK5Q.jpeg" /><figcaption>Image Property of Author — Create with Nightcafe</figcaption></figure><h3>Improving LLM Inference Speeds on CPUs with Model Quantization</h3><h4>Discover how to significantly improve inference latency on CPUs using quantization techniques for bf16, int8, and int4 precisions</h4><p>One of the most significant challenges the AI space faces is the need for computing resources to host large-scale production-grade LLM-based applications. At scale, LLM applications require redundancy, scalability, and reliability, which have historically been only possible on general computing platforms like CPUs. Still, the prevailing narrative today is that CPUs cannot handle LLM inference at latencies comparable with high-end GPUs.</p><p>One open-source tool in the ecosystem that can help address inference latency challenges on CPUs is the <a href="https://www.intel.com/content/www/us/en/developer/tools/oneapi/optimization-for-pytorch.html">Intel® Extension for PyTorch*</a> (IPEX), which provides up-to-date feature optimizations for an extra performance boost on Intel hardware. IPEX delivers a variety of easy-to-implement optimizations that make use of hardware-level instructions. This tutorial will dive into the theory of model compression and the out-of-the-box model compression techniques IPEX provides. These compression techniques directly impact LLM inference performance on general computing platforms, like Intel 4th and 5th-generation CPUs.</p><h3>Inference Latency in Application Development</h3><p>Second only to application safety and security, inference latency is one of the most critical parameters of an AI application in production. Regarding LLM-based applications, latency or throughput is often measured in tokens/second. As illustrated in the simplified inference processing sequence below, tokens are processed by the language model and then de-tokenized into natural language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RIf08bgcZG0K9mw3Yh2c5g.gif" /><figcaption>GIF 1. of inference processing sequence — Image by Author</figcaption></figure><p>Interpreting inference this way can sometimes lead us astray because we analyze this component of AI applications in abstraction of the traditional production software paradigm. Yes, AI apps have their nuances, but at the end of the day, we are still talking about transactions per unit of time. If we start to think about inference as a transaction, like any other, from an application design point of view, the problem becomes less complex. For example, let’s say we have a chat application that has the following requirements:</p><ul><li>Average of <strong>300 user sessions</strong> per hour</li><li>Average of <strong>5 transactions</strong> (LLM inference requests) per user per session</li><li>Average <strong>100 tokens</strong> generated per transaction</li><li>Each session has an average of <strong>10,000ms (10s) overhead</strong> for user authentication, guardrailing, network latency, and pre/post-processing.</li><li>Users take an average of <strong>30,000ms (30s) to respond</strong> when actively engaged with the chatbot.</li><li>The average total active <strong>session time goal is 3</strong> minutes or less.</li></ul><p>Below, you can see that with some simple napkin math, we can get some approximate calculations for the required latency of our LLM inference engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zp9qM6qBFosRLCQG5-de9Q.png" /><figcaption>Figure 1. A simple equation to calculate the required transaction and token latency based on various application requirements. — Image by Author</figcaption></figure><p>Achieving required latency thresholds in production is a challenge, especially if you need to do it without incurring additional compute infrastructure costs. In the remainder of this article, we will explore one way that we can significantly improve inference latency through model compression.</p><h3>Model Compression</h3><p>Model compression is a loaded term because it addresses a variety of techniques, such as model quantization, distillation, pruning, and more. At their core, the chief aim of these techniques is to reduce the computational complexity of neural networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jq0mIHkHT0LKAWbZN7ELoQ.gif" /><figcaption>GIF 2. Illustration of inference processing sequence — Image by Author</figcaption></figure><p>The method we will focus on today is model quantization, which involves reducing the byte precision of the weights and, at times, the activations, reducing the computational load of matrix operations and the memory burden of moving around larger, higher precision values. The figure below illustrates the process of quantifying fp32 weights to int8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cv0izqxOGSQT6-bZS72bSA.png" /><figcaption>Fig 2. Visual representation of model quantization going from full precision at FP32 down to quarter precision at INT8, theoretically reducing the model complexity by a factor of 4. — Image by Author</figcaption></figure><p>It is worth mentioning that the reduction of complexity by a factor of 4 that results from quantizing from fp32 (full precision) to int8 (quarter precision) does not result in a 4x latency reduction during inference because inference latency involves more factors beyond just model-centric properties.</p><p>Like with many things, there is no one-size-fits-all approach, and in this article, we will explore three of my favorite techniques for quantizing models using IPEX:</p><h4><strong>Inference at bf16 or fp32</strong></h4><p>This technique quantizes the weights in the neural network down to a user defined precision. This technique is ideal for smaller models, like the &lt;1B LLMs of the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bB3fjRwe5lDPkv5UD3qLGw.png" /><figcaption>Fig 3. Simple illustration of bf16/fp32, showing FP32 weights in orange and half-precision quantized bf16 weights in green. — Image by Author</figcaption></figure><p>The implementation is quite straightforward: using hugging face transformers, a model can be loaded into memory and optimized using the IPEX llm-specific optimization function ipex.llm.optimize(model, dtype=dtype) by setting dtype = torch.bfloat16, we can activate the half-prevision inference capability, which improves the inference latency over full-precision (fp32) and stock.</p><pre>import sys<br>import os<br>import torch<br>import intel_extension_for_pytorch as ipex<br>from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline<br><br># PART 1: Model and tokenizer loading using transformers<br>tokenizer = AutoTokenizer.from_pretrained(&quot;Intel/neural-chat-7b-v3-3&quot;)<br>model = AutoModelForCausalLM.from_pretrained(&quot;Intel/neural-chat-7b-v3-3&quot;)<br><br># PART 2: Use IPEX to optimize the model<br>#dtype = torch.float # use for full precision FP32<br>dtype = torch.bfloat16 # use for half precision inference<br>model = ipex.llm.optimize(model, dtype=dtype)<br><br># PART 3: Create a hugging face inference pipeline and generate results<br>pipe = pipeline(&quot;text-generation&quot;, model=model, tokenizer=tokenizer)<br>st = time.time()<br>results = pipe(&quot;A fisherman at sea...&quot;,  max_length=250)<br>end = time.time()<br>generation_latency = end-st<br><br>print(&#39;generation latency: &#39;, generation_latency)<br>print(results[0][&#39;generated_text&#39;])</pre><p>Of the three compression techniques we will explore, this is the easiest to implement (measured by unique lines of code) and offers the smallest net improvement over a non-quantized baseline.</p><h4><strong>SmoothQuant (int8)</strong></h4><p>This technique addresses the core challenges of quantizing LLMs, which include handling large-magnitude outliers in activation channels across all layers and tokens, a common issue that traditional quantization techniques struggle to manage effectively. This technique employs a joint mathematical transformation on both weights and activations within the model. The transformation strategically reduces the disparity between outlier and non-outlier values for activations, albeit at the cost of increasing this ratio for weights. This adjustment renders the Transformer layers “quantization-friendly,” enabling the successful application of int8 quantization without degrading model quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pVoW8GbSvsfQ0JwcuBhDTA.png" /><figcaption>Fig 4. Simple illustration of SmoothQuant showing weights as circles and activations as triangles. The diagram depicts the two main steps: (1) the application of scaler for smoothing and (2) the quantization to int8 — Image by Author</figcaption></figure><p>Below, you’ll find a simple SmoothQuant implementation — omitting the code for creating the DataLoader, which is a common and well-documented PyTorch principle. SmoothQuant is an accuracy-aware post-training quantization recipe, meaning that by providing a calibration dataset and model you will be able to provide a baseline and limit the language modeling degradation. The calibration model generates a quantization configuration, which is then passed to ipex.llm.optimize() along with the SmoothQuant mapping. Upon execution, the SmoothQuant is applied, and the model can be tested using the .generate() method.</p><pre>import torch<br>import intel_extension_for_pytorch as ipex<br>from intel_extension_for_pytorch.quantization import prepare<br>import transformers<br><br># PART 1: Load model and tokenizer from Hugging Face + Load SmoothQuant config mapping<br>tokenizer = AutoTokenizer.from_pretrained(&quot;Intel/neural-chat-7b-v3-3&quot;)<br>model = AutoModelForCausalLM.from_pretrained(&quot;Intel/neural-chat-7b-v3-3&quot;)<br>qconfig = ipex.quantization.get_smooth_quant_qconfig_mapping()<br><br># PART 2: Configure calibration<br># prepare your calibration dataset samples<br>calib_dataset = DataLoader({Your dataloader parameters})<br>example_inputs = # provide a sample input from your calib_dataset<br>calibration_model = ipex.llm.optimize(<br>  model.eval(),<br>  quantization_config=qconfig,<br>)<br>prepared_model = prepare(<br>  calibration_model.eval(), qconfig, example_inputs=example_inputs<br>)<br>with torch.no_grad():<br>  for calib_samples in enumerate(calib_dataset):<br>    prepared_model(calib_samples)<br>prepared_model.save_qconf_summary(qconf_summary=qconfig_summary_file_path)<br><br># PART 3: Model Quantization using SmoothQuant<br>model = ipex.llm.optimize(<br>  model.eval(),<br>  quantization_config=qconfig,<br>  qconfig_summary_file=qconfig_summary_file_path,<br>)<br><br># generation inference loop<br>with torch.inference_mode():<br>    model.generate({your generate parameters})</pre><p>SmoothQuant is a powerful model compression technique and helps significantly improve inference latency over full-precision models. Still, it requires a little upfront work to prepare a calibration dataset and model.</p><h4><strong>Weight-Only Quantization (int8 and int4)</strong></h4><p>Compared to traditional int8 quantization applied to both activation and weight, weight-only quantization (WOQ) offers a better balance between performance and accuracy. It is worth noting that int4 WOQ requires dequantizing to bf16/fp16 before computation (Figure 4), which introduces an overhead in compute. A basic WOQ technique, tensor-wise asymmetric Round To Nearest (RTN) quantization, presents challenges and often leads to reduced accuracy (<a href="https://huggingface.co/blog/intel-starcoder-quantization">source</a>). However, literature (Zhewei Yao, 2022) suggests that groupwise quantizing the model’s weights helps maintain accuracy. Since the weights are only dequantized for computation, a significant memory advantage remains despite this extra step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qEmqizumc4X5C0UhtsD1hg.png" /><figcaption>Fig 5. Simple illustration of weight-only quantization, with pre-quantized weights in orange and the quantized weights in green. Note that this depicts the initial quantization to int4/int8 and dequantization to fp16/bf16 for the computation step. — Image by Author</figcaption></figure><p>The WOQ implementation below showcases the few lines of code required to quantize a model from Hugging Face with this technique. As with the previous implementations, we start by loading a model and tokenizer from Hugging Face. We can use the get_weight_only_quant_qconfig_mapping() method to configure the WOQ recipe. The recipe is then passed to the ipex.llm.optimize() function along with the model for optimization and quantization. The quantized model can then be used for inference with the .generate() method.</p><pre><br># requirements<br>#intel-extension-for-pytorch==2.2<br>#transformers==4.35.2<br>#torch==2.2.0 <br><br>import torch<br>import intel_extension_for_pytorch as ipex<br>from transformers import AutoTokenizer, AutoModelForCausalLM<br><br># PART 1: Model and tokenizer loading<br>tokenizer = AutoTokenizer.from_pretrained(&quot;Intel/neural-chat-7b-v3-3&quot;)<br>model = AutoModelForCausalLM.from_pretrained(&quot;Intel/neural-chat-7b-v3-3&quot;)<br><br># PART 2: Preparation of quantization config<br>qconfig = ipex.quantization.get_weight_only_quant_qconfig_mapping(<br>  weight_dtype=torch.qint8, # or torch.quint4x2<br>  lowp_mode=ipex.quantization.WoqLowpMode.NONE, # or FP16, BF16, INT8<br>)<br>checkpoint = None # optionally load int4 or int8 checkpoint<br><br># PART 3: Model optimization and quantization<br>model = ipex.llm.optimize(model, quantization_config=qconfig, low_precision_checkpoint=checkpoint)<br><br>inputs = tokenizer(&quot;I love learning to code...&quot;, return_tensors=&quot;pt&quot;).input_ids<br><br># PART 4: Inference output generation<br>with torch.inference_mode():<br>    tokens = model.generate(<br>        inputs,<br>        max_new_tokens=64,<br>)<br><br>print(tokenizer.decode(tokens[0], skip_special_tokens=True))</pre><p>As you can see, WOQ provides a powerful way to compress models down to a fraction of their original size with limited impact on language modeling capabilities.</p><h3>Conclusion and Discussion</h3><p>As an engineer at Intel, I’ve worked closely with the IPEX engineering team at Intel. This has afforded me a unique insight into its advantages and development roadmap, making IPEX a preferred tool. However, for developers seeking simplicity without the need to manage an extra dependency, PyTorch offers <a href="https://pytorch.org/docs/stable/quantization.html">three quantization recipes</a>: Eager Mode, FX Graph Mode (under maintenance), and PyTorch 2 Export Quantization, providing strong, less specialized alternatives.</p><p>No matter what technique you choose, model compression techniques will result in some degree of language modeling performance loss, albeit in &lt;1% in many cases. For this reason, it’s essential to evaluate the application’s fault tolerance and establish a baseline for model performance at full (FP32) and/or half-precision (BF16/FP16) before pursuing quantization.</p><p>In applications that leverage some degree of in-context learning, like Retrieval Augmented Generation (RAG), model compression might be an excellent choice. In these cases, the mission-critical knowledge is spoon-fed to the model at the time of inference, so the risk is heavily reduced even with low-fault-tolerant applications.</p><p>Quantization is an excellent way to address LLM inference latency concerns without upgrading or expanding compute infrastructure. It is worth exploring regardless of your use case, and IPEX provides a good option to start with just a few lines of code.</p><p><strong>A few exciting things to try would be:</strong></p><ul><li>Test the sample code in this tutorial on the Intel Developer Cloud’s <a href="https://console.idcservice.net/training">free Jupyter Environment</a>.</li><li>Take an existing model that you’re running on an accelerator at complete precision and test it out on a CPU at int4/int8</li><li>Explore all three techniques and determine which works best for your use case. Make sure to compare the loss of language modeling performance, not just latency.</li><li><a href="https://github.com/intel/ai-innovation-bridge/blob/master/workshops/ai-workloads-with-huggingface/6%20-%20Uploading%20and%20Sharing%20Models%20on%20Hugging%20Face%20Hub%20with%20Intel%20Optimizations.ipynb">Upload your quantized model to the Hugging Face Model Hub</a>! If you do, let me know — I’d love to check it out!</li></ul><p><strong><em>Thank you for reading! Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28aefb495657" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/improving-llm-inference-latency-on-cpus-with-model-quantization-28aefb495657">Improving LLM Inference Latency on CPUs with Model Quantization</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed Fine-Tuning of Stable Diffusion with CPUs on AWS]]></title>
            <link>https://awstip.com/distributed-fine-tuning-of-stable-diffusion-with-cpus-on-aws-0e8eabc9f50c?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e8eabc9f50c</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[stable-diffusion]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Wed, 20 Dec 2023 21:41:46 GMT</pubDate>
            <atom:updated>2023-12-22T16:09:52.923Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s47byC_UVYMh15vOVrM_ww.jpeg" /><figcaption>Image Property of Author</figcaption></figure><h4>Learn how to use Hugging Face* Accelerate on Amazon Web Services (AWS)* to fine-tune Stable Diffusion</h4><p>Today’s AI landscape primarily revolves around language modeling. Still, we should not forget that diffusion models helped popularize generative AI by capturing the imagination of the masses with their text-to-image capabilities. Diffusion models are not based on transformer architectures, but like transformer architectures, they can be fine-tuned to generate content using a relevant set of images.</p><p>This article will focus on textural inversion, a technique requiring only a few sample images. We will explore a handful of optimizations leveraging 4th Generation Intel® Xeon® processors on AWS. By the end of this article, you will have fine-tuned a stable diffusion model using Hugging Face’s Diffusers and Accelerate and the <a href="https://www.intel.com/content/www/us/en/developer/tools/oneapi/optimization-for-pytorch.html">Intel Extension of PyTorch</a>.</p><h3>What are Stable Diffusion Models?</h3><p>These models are a type of diffusion model, a class of generative models. They work by gradually transforming a random noise distribution into a data distribution of interest (like images) over a series of steps. This process is akin to starting with a canvas of random noise and slowly refining it into a coherent image.</p><p>Stable Diffusion models are a combination of three models — a variational autoencoder (VAE), a text encoder like contrastive language–image retraining (CLIP), and a U-Net — and the specific role of each in the training process is largely accurate. Here’s a breakdown:</p><ol><li><strong>Variational Autoencoder (VAE):</strong> The VAE encodes images into a lower-dimensional latent space and decodes these latent representations back into images. This compression and reconstruction allow the diffusion process to operate in a more efficient and manageable latent space rather than directly on high-dimensional image data.</li><li><strong>Text Encoder (such as CLIP): </strong>The text encoder, like CLIP (Contrastive Language–Image Pretraining), provides a way to encode textual descriptions into a latent space comparable to the image encodings. This enables the model to understand and relate text descriptions to images, facilitating text-to-image generation.</li><li><strong>U-Net:</strong> The U-Net is the core component of the diffusion process. During diffusion training, the U-Net is trained to gradually denoise images. It learns to reverse the process of adding noise to an image (done during the forward diffusion process), thus generating an image from a noisy state. In the context of Stable Diffusion, only the U-Net is trained during the diffusion training phase.</li></ol><p>The training process involves the U-Net learning to create images from noise, guided by the latent encodings from the VAE and the text descriptions from the CLIP-like model. The VAE and text encoder are pre-trained models and are not typically trained further during the diffusion training process. They provide the context and structure for the U-Net to generate coherent and relevant images based on textual input.</p><h3>Distributed Training</h3><p>Distributed training of AI models refers to training a machine learning model across multiple devices (Figure 1), such as CPUs, GPUs, or ASICs, to improve training efficiency and speed up the process. This is particularly important for large and complex models, such as deep neural networks, which require enormous computational resources and can take considerable time to train.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8JKqFctt8xhB5aWRYa_ZNg.png" /><figcaption>Figure 1. Illustration of a basic distributed training system — image by author</figcaption></figure><p>Imagine a construction site where multiple workers are building a large structure, such as a pyramid (Figure 2). Each worker is responsible for laying bricks in a specific place, and they need to work together to ensure that the pyramid is built correctly and efficiently. The project manager is in charge of coordinating their efforts, making sure that everyone knows what to do and when to do it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*lCJfVe7n0R8_Z2MYaPDZZw.gif" /><figcaption>Figure 2. Illustration of construction worker analogy for distributed training — image by author</figcaption></figure><p>In this analogy, the construction workers represent the multiple nodes used in distributed training, and the project manager represents the centralized server or communication system that synchronizes their updates. Just like the workers need to cooperate to build the pyramid, the nodes need to work together to train the AI model. The project manager ensures that everyone is on the same page, similar to how a centralized server or communication system (e.g. oneCCL) ensures that all devices have an updated model version. By dividing the work and coordinating their efforts, the construction workers can build the pyramid faster, and the devices can train the AI model more efficiently.</p><h3>The Stack</h3><p>The technical stack in this article is quite simple thanks to Hugging Face Accelerate. Previously, people required tools like Kubernetes to manage their distributed training clusters. However, Accelerate abstracts away many complexities and makes it easier to set up a system with just a few simple commands. Depending on the scale of your distributed training workload, you might consider other tools, but Accelerate is sufficient for this example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2FNeaFj72LGMyit6OVUn1g.png" /></figure><p>For compute, we will leverage AWS EC2 to access m7i (4th Generation Xeon) CPU instances. These instances have <a href="https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/advanced-matrix-extensions/overview.html">Intel Advanced Matrix Extensions (AMX)</a>, which optimize memory management and matrix operations for deep learning workloads, acting like a small accelerator on the CPU. We will access these optimizations by invoking the Intel Extension for PyTorch.</p><p>Overall, this technical stack simplifies setting up a managed cluster for distributed training, making it easier for developers to focus on model performance rather than configuring complex tools.</p><h3>Stable Diffusion Fine-Tuning Walkthrough</h3><p>We will start by setting up our infrastructure, followed by setting up the distributed fine-tuning parameters, and conclude with executing the training process. All of the code for this tutorial can be found <a href="https://github.com/intel/intel-cloud-optimizations-aws/tree/main/distributed-training/stable_diffusion">here</a>.</p><h4>1. AWS prerequisites</h4><p>Before proceeding, ensure you have an AWS account and the necessary permissions to launch EC2 instances, create Amazon Machine Images (AMIs), create security groups, and create S3 storage buckets.</p><p>In this example, we use three <a href="https://aws.amazon.com/ec2/instance-types/m6i/"><em>m7i.4xlarge</em> EC2 instances</a> with Ubuntu 22.04 and 250 GB of storage each. To start, you must launch a single EC2 instance and connect to it using SSH. You can do so from the AWS console with the instructions found <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-instance-wizard.html">here</a>.</p><p>If you are using a 4th Generation Xeon CPU, you can verify that you have the AMX instruction set by running:</p><pre>lscpu | grep amx</pre><p>If AMX is available, you should see the following flags:</p><pre>amx_bf16 amx_tile amx_int8</pre><p>These flags indicate that the AMX instructions are available on your system, which is essential for leveraging mixed precision training and using bfloat16.</p><p><em>Note: For now, Intel AMX is currently available only in 4th and 5th Gen Xeon processors</em></p><h4>2. Clone the repository and install dependencies</h4><p>To simplify the setup of this workload, we have provided a <em>Make </em>script with various targets (instructions) that can be executed with a single command.</p><p>Start by installing <em>Make</em></p><pre>sudo apt install make</pre><p>Clone the cloud optimization module repository</p><pre>git clone https://github.com/intel/intel-cloud-optimizations-aws.git</pre><p>Navigate to the setup folder</p><pre>cd intel-cloud-optimizations-aws/distributed-training/stable_diffusion/</pre><p>Install miniconda by running</p><pre>make install-miniconda</pre><p>Source bash profile to activate base conda env</p><pre>source ~/.bashrc</pre><p>Run make instructions below. We’ve created this instruction to make setting up the environment in the primary node as easy as possible. This command will install PyTorch, Accelerate, Intel Extension for Pytorch, oneCCL, and other critical dependencies.</p><pre>make setup-stable-diffusion-icom-master</pre><h4>3. Adding optimizations to training code</h4><p>We recommend using a text editor for this part if you want to practice making the changes yourself. However, we have already done this for you, and you can use the textual_inversion_icom.py file that is part of the provided code.</p><p>If you are starting from scratch with the original textual_inversion.py script from diffusers or another script, you will enable the Intel Extension for PyTorch optimizations by adding:</p><pre>import intel_extension_for_pytorch as ipex<br>unet = ipex.optimize(unet, dtype=weight_dtype)<br>vae = ipex.optimize(vae, dtype=weight_dtype)</pre><p>To test on a single node, we can run the single-node-test M<em>ake </em>target again. Ensure you are in the<em> setup/</em> directory when running this:</p><pre>make single-node-test</pre><p>When fine-tuning is completed, the single node only runs five steps and should save the learned_embeds.safetensors in the <em>textual_inversion_output</em> folder. Once we successfully test on a single node, we can begin setting up the rest of the infrastructure for the distributed training.</p><h4>3. Setting up amazon machine image (AMI) for distributed training</h4><p>In this section, we will set up the infrastructure on AWS. At the core of this process is the AMI, which will allow us to “copy and paste” our primary node to create the additional nodes in our system. Before creating this AMI, we will complete the configuration of our primary node by executing the following steps.</p><p>First, Let’s install <em>net-tools</em> that will allow us to retrieve the I_MPI_HYDRA_IFACE environment variable.</p><pre>sudo pat install net-tools<br>ifconfig -a | grep enp</pre><p>Run the following to establish the required environment variables.</p><pre>export I_MPI_HYDRA_IFACE=&lt;replace with value from previous step&gt;<br>oneccl_bindings_for_pytorch_path=$(python -c &quot;from oneccl_bindings_for_pytorch import cwd; print(cwd)&quot;)<br>source $oneccl_bindings_for_pytorch_path/env/setvars.sh<br>export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libiomp5.so<br>export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so<br>export CCL_ATL_TRANSPORT=ofi<br>export CCL_WORKER_COUNT=1<br><br>export MODEL_NAME=&quot;runwayml/stable-diffusion-v1-5&quot;<br>export DATA_DIR=&quot;&lt;path dicoo dataset&gt;</pre><p>Also, delete the textual_inversion_output folder, created during our test run, by running:</p><pre>rm -rf textual_inversion_output</pre><p>Passwordless SSH enables communication between nodes and requires several steps, which will be distributed throughout the next few sections. The first step will be to add a custom ssh key to the primary node. If the following approach does not accommodate your workspace, use another method.</p><ol><li>Locate your AWS EC2 instance key. This is the key that you specified when launching the instance. It should have a “.pem” extension</li><li>Open the file in a text editor</li><li>Copy the output of the command, which is your key.</li><li>SSH into the remote host where you want to add your key.</li><li>Once logged in, navigate to the ~/.ssh directory on the remote host if it exists. If the directory does not exist, you can create it: mkdir -p ~/.ssh</li><li>Within the ~/.ssh directory on the remote host, create a file named id_rsa: nano ~/.ssh/id_rsa</li><li>Paste the previously copied public key into this file, ensuring it’s on a new line.</li><li>Save the changes and exit the text editor.</li><li>Adjust permissions to file chmod 600 ~/.ssh/id_rsa</li></ol><p><strong>Create an AMI<br></strong>Start by creating an Amazon Machine Image (AMI) from the existing instance where you have successfully run the fine-tuning on a single system. This AMI will capture the entire setup, including the dependencies, configurations, codebase, and dataset. To create an AMI, refer to <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html#:~:text=EBS%20volumes.-,Create%20a%20Linux%20AMI%20from%20an%20instance,-You%20can%20create">Create a Linux AMI from an instance</a>.</p><p><strong>Security Group<br></strong> While waiting for the AMI creation, let’s continue by creating a security group that enables communication among the member nodes. This security group should be configured to allow inbound and outbound traffic on the necessary ports for effective communication between the primary node and the worker nodes. The easiest place to do this is from the AWS console.</p><p>In the security group configuration, ensure that you have allowed <em>all</em> traffic to originate from the security group itself (Figure 1). This setting allows seamless communication between the instances within the security group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wrz6M6l_LaOSFCdp" /><figcaption>Figure 3. security group configuration to enable passwordless SSH — image by author</figcaption></figure><p>Setting up the security group ensures that all necessary traffic can flow between the primary and secondary nodes during distributed training.</p><h4>4. Setting up secondary Instances</h4><p>Now, we will set up our secondary instances using the AMI we just created. These will serve as the rank 1 and 2 nodes in our distributed infrastructure. We will leverage the aws-cli; if you don’t already have the aws-cli installed, follow the instructions on this <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">site</a> to install it.</p><p><strong>Launch new instances<br></strong>Use the created AMI to launch new instances, specifying the desired number of instances based on the number of systems you want to use for distributed training. This ensures that all the instances have the same environment and setup. Two options for initiating new EC2 instances are: using the AWS console or aws-cli. If you have aws-cli configured, you can launch instances by executing the following command:</p><pre>aws ec2 run-instances --image-id ami-xxxxxxxxx --count 2 --instance-type m7i.4xlarge --key-name &lt;name of preferred ec2 key&gt; --security-group-ids sg-xxxxxxxxx --subnet-id subnet-xxxxxxxxx</pre><p>After completing the public key above to all nodes, verify that you can connect using the key pair. Run the following from the node you have been working from (primary node).</p><pre>ssh &lt;username&gt;@&lt;private-ip-address&gt;</pre><p>Next, to continue setting up the cluster, you will need to edit the SSH configuration file located at ~/.ssh/config on the master node (if it doesn&#39;t exist, you can create one). The configuration file should look like this:</p><pre>Host 10.*.*.*<br>   StrictHostKeyChecking no<br><br>Host node1<br>    HostName 10.0.xxx.xxx<br>    User ubuntu<br><br>Host node2<br>    HostName 10.0.xxx.xxx<br>    User ubuntu</pre><p>In the host section, you will put the Private IPv4 addresses. You can get these for each node from the AWS console OR by running hostname -i when connected to the corresponding node.</p><p>The StrictHostKeyChecking no line disables strict host key checking, allowing the master node to SSH into the worker nodes without prompting for verification.</p><p>With these settings, you can check your passwordless SSH by executing ssh node1 or ssh node2 to connect to any node without any additional prompts.</p><p>Additionally, on the master node, you will create a host file (~/hosts) that includes the names of all the nodes you want to include in the training process, as defined in the SSH configuration above. Use localhost for the master node itself as you will launch the training script from the master node. You can stick this file in your working directory. The hosts file will look like this:</p><pre>localhost<br>node1<br>node2</pre><p>This setup will allow you to seamlessly connect to any node in the cluster for distributed training.</p><h4>4. Preparing for distributed training</h4><p>Now that we have this running in a single system let’s try to run it on multiple systems. To prepare for distributed training and ensure a consistent setup across all systems, follow these steps:</p><ol><li>Configure Accelerate: Activate your conda environment. If you did not change its name in the make file, it should be called “diffuser_icom” e.g. conda activate diffuser_icom</li><li>We need to prepare a new “accelerate” config for multi-CPU setup. But before setting up the multi-CPU environment, ensure you have your machine’s IP address handy. To obtain it, run the following command hostname -i</li><li>With the IP address ready, execute the following command to generate the new accelerate config for the multi-CPU setup: accelerate config</li><li>You will be prompted with several questions when configuring the multi-CPU setup using accelerate config. Select the appropriate answers based on your environment. Here&#39;s a step-by-step guide on how to proceed:</li></ol><pre>In which compute environment are you running?<br>Please select a choice using the arrow or number keys, and selecting with enter<br> ➔  This machine<br>    AWS (Amazon SageMaker)</pre><p>Choose multi-CPU as the type of machine for our setup.</p><pre>Which type of machine are you using?<br>Please select a choice using the arrow or number keys, and selecting with enter<br>    No distributed training       <br> ➔  multi-CPU<br>    multi-XPU<br>    multi-GPU<br>    multi-NPU<br>    TPU</pre><p>Next, you can enter the number of instances you will be using. For example, here we have 3 (including the master node).</p><pre>How many different machines will you use (use more than 1 for multi-node training)? [1]: 3</pre><p>Concerning the rank, since we are initially running this from the master node, enter 0. For each machine, you will need to change the rank accordingly.</p><pre>What is the rank of this machine?<br>Please select a choice using the arrow or number keys, and selecting with enter<br> ➔  0<br>    1<br>    2</pre><p>Next, you will need to provide the private IP address of the machine where you are running the accelerate launch command (main node - likely the same node you have been working on this whole time), that we found earlier with hostname -i.</p><pre>What is the IP address of the machine that will host the main process?   </pre><p>Next, you can enter the port number to be used for communication. A commonly used port is 29500, but you can choose any available port.</p><pre>What is the port you will use to communicate with the main process?   </pre><p>The following prompt is actually about CPU sockets. Generally, each machine will have only 1 CPU socket. However, in the case of bare metal instances, you may have 2 CPU sockets per instance. Enter the appropriate number of sockets based on your instance configuration.</p><pre>How many CPU(s) should be used for distributed training?</pre><p>After completing the configuration, you will be ready to launch the multi-CPU fine-tuning process. The final output should look something like this:</p><pre>------------------------------------------------------------------------------------------------------------------------------------------<br>In which compute environment are you running?<br>This machine<br>------------------------------------------------------------------------------------------------------------------------------------------<br>Which type of machine are you using?<br>multi-CPU<br>How many different machines will you use (use more than 1 for multi-node training)? [1]: 3<br>------------------------------------------------------------------------------------------------------------------------------------------<br>What is the rank of this machine?<br>0<br>What is the IP address of the machine that will host the main process? xxx.xxx.xxx.xxx<br>What is the port you will use to communicate with the main process? 29500<br>Are all the machines on the same local network? Answer `no` if nodes are on the cloud and/or on different network hosts [YES/no]: nossh no<br>What rendezvous backend will you use? (&#39;static&#39;, &#39;c10d&#39;, ...): static<br>Do you want to use Intel PyTorch Extension (IPEX) to speed up training on CPU? [yes/NO]:yes<br>Do you wish to optimize your script with torch dynamo?[yes/NO]:NO<br>How many CPU(s) should be used for distributed training? [1]:1<br>------------------------------------------------------------------------------------------------------------------------------------------<br>Do you wish to use FP16 or BF16 (mixed precision)?<br>bf16</pre><p>This process should have generated a new config file named multi_config.yaml in the .cache folder. You now have 2 options:</p><ul><li><strong>Option 1:</strong> Repeat the accelerate configure process on each node by connecting into the worker nodes and running accelerate config</li><li><strong>Option 2: </strong>Copy the contents of the hugging face accelerate cache to the other nodes. You would need to change the machine rank in the yaml file to correspond to the rank of each machine in the distributed system.</li></ul><h4>5. Fine-tuning on multiple CPUs</h4><p>Finally, it’s time to run the fine-tuning process on multi-CPU setup. Make sure you are connected to your main machine (rank 0) and in the “./stable_diffusion/” directory. Run the following command be used to launch distributed training:</p><pre>mpirun -f ./hosts -n 3 -ppn 1 accelerate launch textual_inversion_icom.py --pretrained_model_name_or_path=&quot;runwayml/stable-diffusion-v1-5&quot; --train_data_dir=&quot;./dicoo/&quot; --learnable_property=&quot;object&quot;   --placeholder_token=&quot;&lt;dicoo&gt;&quot; --initializer_token=&quot;toy&quot; --resolution=512  --train_batch_size=1  --seed=7  --gradient_accumulation_steps=1 --max_train_steps=50 --learning_rate=2.0e-03 --scale_lr --lr_scheduler=&quot;constant&quot; --lr_warmup_steps=3 --output_dir=./textual_inversion_output --mixed_precision bf16 --save_as_full_pipeline</pre><p>The training will start immediately. You can ssh into the other instances and run htop to see the thread utilization across each of the nodes. In this example, we only run 50 training steps — this is something you should seek to optimize in your implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*tC6iErcRIafpTw-Q8U8bKA.gif" /><figcaption>Figure 4. Short video of fine-tuning execution and evaluation of thread utilization across all three nodes in distributed system of m7i.4xlarge AWS EC2 instances — video by author</figcaption></figure><p>Upon completing the training, a new folder will be created in your working directory which will contain the trained model in <em>safetensor </em>format.</p><h3>Conclusion and Discussion</h3><p>In this article, we explored the use of AWS EC2 instances with 4th Gen Intel Xeon CPUs on AWS to fine-tune a stable diffusion model. Distributed training provides various benefits, most of which center on more scalable and efficient training cycles for large deep learning models. In our simple example, we provide the code and steps to configure a distributed compute system on the AWS cloud with optimizations from Hugging Face and the Intel Extension for PyTorch.</p><p>The combination of flexible compute in the cloud, Hugging Face simple abstractions, and optimizations from Intel, yield a simpler workflow for training complex models.</p><p><strong>A few exciting things to try would be:</strong></p><ul><li>Adjust the fine-tuning hyperparameters to optimize the model’s performance.</li><li>Test different m7i instances sizes and number of nodes in the distributed system to define optimal scaling efficiency for your workload.</li><li>Use a different dataset, that aligns with your business or academic project.</li></ul><p><strong><em>Thank you for reading! Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e8eabc9f50c" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/distributed-fine-tuning-of-stable-diffusion-with-cpus-on-aws-0e8eabc9f50c">Distributed Fine-Tuning of Stable Diffusion with CPUs on AWS</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieval Augmented Generation (RAG) Inference Engines with LangChain on CPUs]]></title>
            <link>https://medium.com/data-science/retrieval-augmented-generation-rag-inference-engines-with-langchain-on-cpus-d5d55f398502?source=rss-e49cc416a8ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5d55f398502</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Eduardo Alvarez]]></dc:creator>
            <pubDate>Tue, 05 Dec 2023 06:02:56 GMT</pubDate>
            <atom:updated>2023-12-05T06:02:56.846Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZsYzboxzqR4B9IQRI6rw3A.jpeg" /><figcaption>Created with Nightcafe — Property of Author</figcaption></figure><h4>Exploring scale, fidelity, and latency in AI applications with RAG</h4><p>While Retrieval Augmented Generation (RAG) is extensively covered, particularly in its application to chat-based LLMs, in this article we aim to view it from a different perspective and analyze its prowess as a powerful operational tool. We will also provide a useful hands-on example to get practical experience with RAG-based applications. By the end of the article, you’ll develop a unique vantage point on RAG — understanding its role and potential in scalable inference within production-scale LLM deployments.</p><h3>But first, let’s refresh our understanding of inference</h3><p>Inference is the process that transforms data into predictions. This component of the ML lifecycle is often cradled by data pipelines that manage pre-processing and post-processing tasks. Let’s evaluate a practical example, consider a music streaming service’s recommendation system, as shown in Figure 1. When a user visits the streaming platform, an intelligently curated list of top 10 songs is presented in the application’s interface. The recommendation system responsible for this list relies on a trained model and robust data pipelines to ensure a high-quality result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*OzaOrxlbtSvKcgCX0YuPZQ.png" /><figcaption>Figure 1. Simple diagram illustrating a recommendation system supporting “top 10 recommended song list” functionality — image by author</figcaption></figure><p>The pre-processing stage, represented by yellow boxes in our diagram, is crucial for ensuring that the model’s prediction closely aligns with the user’s unique taste. Starting with the last 250 songs played by the user, the pipeline processes the data and generates a set of contextual features before passing it to a trained model for inference. The inference step predicts what this user might like and yields an output that passes to a post-processing stage (illustrated in orange). In this final step, the model’s top recommendations are enriched with additional metadata — album art, song titles, artist names, and their ranking. This information is then displayed on the user’s interface for consumption.</p><p>In the workflow described above, it’s clear how proximal the inference step is to the user in the application topology. Unlike other AI lifecycle components like data collection and model optimization, which tend to operate in the background, the inference engine is in the frontline, interacting closely with the user interface (UI) and user experience (UX).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Jn2Ow3fduEQEM_zGOs-UQ.png" /><figcaption>Figure 2. An illustration of various components of an AI system and their proximity to user experience. — Image by author</figcaption></figure><p>We can use the diagram above (Figure 2) to illustrate the proximity of various components of the AI lifecycle to the user. While many components like data collection and annotation sit “behind the scenes,” the inference engine stands as a critical bridge between the AI’s internal processes and what end-users are exposed to. It’s not just another backend mechanism, it is a core part of the user’s tangible experience with the application.</p><p>Given this critical role in shaping user experience, it is essential that the inference engine — encompassing the inference process and its peripheral components like pre/post-processing, API management, and compute management — operates flawlessly. To establish boundary conditions for inference engine quality, I introduce the “Inference Quality (IQ) Triangle,” depicted in Figure 3. This qualitative figure highlights three pivotal aspects to focus on when enhancing the performance of an inference workload:</p><ul><li><strong>Latency: </strong>if the inference engine takes less time to yield a response, it reduces overhead for the application and leads to a better user experience.</li><li><strong>Fidelity: </strong>inference needs to provide answers that users can trust and feel confident in. This includes but is not limited to ensuring high accuracy of responses and reducing hallucinations.</li><li><strong>Scalability:</strong> As the load on the AI system fluctuates, the ability to scale infrastructure is key to optimizing cost and enabling the right-sizing of computing resources.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xra-1dq7GYRYDf6bd2l7qQ.png" /><figcaption>Figure 3. A simplistic model for aligning three key elements of high-quality inference engines. A strong focus on operation excellence where AI systems need to scale while maintaining low latency and returning high-fidelity insights. — Image by author</figcaption></figure><p>As we progress through the article, we will reference the IQ Triangle to dive deeply into how these three components — latency, fidelity, and scale — align well with the RAG workload.</p><h3>Brief Introduction to Retrieval Augmented Generation</h3><p>Retrieval augmented generation, also known as RAG, is a technique introduced initially by Piktus et al. (2021) in <em>Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks</em> and has since been adapted in various frameworks and applications. RAG falls under the category of in-context learning techniques, which focus on providing additional knowledge to pre-trained models in an effort to augment the quality of their responses.</p><p>The hallmark of RAG is in the intelligent retrieval of additional information from relevant data sources, typically vector databases using algorithms like similarity search. The retrieved data is combined with the user’s query, enriching the input provided to the generative model. A standard RAG workflow is depicted in Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fBIu-VlIa3HgvYlmLd9mSQ.png" /><figcaption>Figure 4. A simple RAG workflow diagram — Image by author</figcaption></figure><p>To grasp RAG’s real value, let’s consider a practical scenario: a financial analyst at a major corporation (Figure 5) is grappling with the task of building a quarterly earnings report. Executing this task in a traditional fashion would be a time-intensive endeavor. LLM-based applications offer significant efficiency improvements, but there’s a catch — the need for up-to-date, proprietary information, which isn’t accessible at the time of training foundational open-source models. This could be partially resolved by fine-tuning but the rapid pace of business operations means this process would require continuous fine-tuning to keep models current.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VV_6w9AssmuRoXE8_01Xzw.png" /><figcaption>Figure 5. Adapted version of Figure 6. In this adaptation, a scenario involving a financial analyst’s use of RAG is represented to illustrate the workflow’s validity in building RAG systems based on proprietary data. — Image by author</figcaption></figure><p>RAG tackles these challenges by retrieving relevant, live data, allowing models to be dynamically updated with the latest information. This puts pressure on the quality of the underlying database, but at least data management is more empirical and predictable than LLM hallucinations and neural network extrapolation. As an additional bonus, this approach safeguards sensitive data within the organization’s data infrastructure.</p><p>Many applied AI engineers agree that there should be a shift toward a hybrid strategy focusing on periodic fine-tuning and robust RAG pipelines. In practice, this strategy experiences improved alignment with domain-specific tasks and increases model relevance in applications with fast-evolving data environments.</p><p><em>In the hands-on example, you can toggle RAG on/off to see the impacts of pre-trained model response quality with and without the context provided by the intelligent retrieval mechanisms. See Figure 10.</em></p><h3>Operational RAG Systems in Support of Quality Inference Engines</h3><p>Now that we have a foundational understanding of RAG and its role in LLM-based applications, we will focus on implementing these systems’ practical and operational aspects.</p><p>As promised, let’s revisit the IQ Triangle (Figure 3) which underscores three vital aspects of high-quality operational inference engines. We will analyze the opportunity to address all three of these aspects — scalability, latency, and fidelity — using the stack illustrated below (Figure 6), which focuses on an inference engine composed of RAG pipelines and heavily optimized models running on CPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tW06G0haX0XMYkz2cLeDiw.png" /><figcaption>Figure 6. Proposed Inference Engine stack — Image my author</figcaption></figure><h4>RAG’s Architectural Benefits</h4><p>RAG-based applications bring significant architectural benefits. From a <strong>scalability </strong>perspective, all data-centric components of the pipeline converge on a single (or few) vector databases (Figure 7), allowing the fresh data benefits of RAG to scale well with increasing/decreasing user requests. This unified approach can significantly improve the <strong>fidelity </strong>of responses to domain-specific tasks while greatly simplifying data governance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQxF4XN388rQa5b7UsYyCw.png" /><figcaption>Figure 7. Simple Data Flow diagram showing the flow of data across a RAG-based AI System — Image by author</figcaption></figure><h4>Optimized Models: Efficiency and Performance</h4><p>Models can achieve smaller computational and environmental footprints through model compression and parameter-efficient fine-tuning techniques. While fine-tuning can help tailor models to specific tasks, enhancing their predictive accuracy (<strong>fidelity</strong>), compression methods like quantization can shrink model sizes, significantly improving inference <strong>latency</strong>. These lean and tuned models are easier to deploy in the data center and enable AI applications at the edge, opening the door for various innovative use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8lSHgoc_UJ9UfcZZGgrAXg.png" /><figcaption>Figure 8 — Image by Author</figcaption></figure><h4>CPUs in Support of RAG</h4><p>Regarding workflows involving complex logic, like RAG, CPUs stand out for their ubiquity and cost-efficiency. This enables improved scale since almost any organization can access enterprise-grade CPUs in the cloud, unlike specialized accelerators, which are harder to come by.</p><p>Modern CPUs also come equipped with low-level optimizations — take, for instance, Intel Advanced Matrix Extensions in their 4th Generation Xeon Processors — which improve memory management and matrix operations in deep learning training and inference phases. Their support for lower precision data types (such as bf16 and int8) makes them well-suited for achieving low <strong>latency </strong>during inference.</p><p>Furthermore, the compatibility of CPUs with the multiple components of a RAG pipeline (Figure 9), including vector databases and intelligent search (for example, similarity search), streamlines infrastructure management, making <strong>scaled </strong>deployment more straightforward and efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q2dj6aiq8QeRrglH9KFiNQ.png" /><figcaption>Figure 9. Adaptation of Figure 7. Showcases the ability of CPUs to support various parts of the RAG system. — Image by author</figcaption></figure><p><em>Before moving on, I must disclose my affiliation with Intel and the product used below. As a Senior AI Engineer at Intel, the following hands-on sample is run on the Intel Developer Cloud (IDC). We will use IDC as </em>a free and convenient way to access Compute to get practical experience<em> with the concepts described in this article.</em></p><h3>Hands-On Example: Implementing RAG with LangChain on the Intel Developer Cloud (IDC)</h3><p>To follow along with the following hands-on example, create a free account on the <a href="https://bit.ly/3sTXnHt">Intel Developer Cloud</a> and navigate to the “Training and Workshops” page. Under the <em>Gen AI Essentials</em> section, select <strong>Retrieval Augmented Generation (RAG) with LangChain </strong>option. Follow the instructions on the webpage to launch a JupyterLab window and automatically load the notebook with all of the sample code.</p><p>The notebook includes detailed docstrings and descriptions of the code. This article will discuss the high-level mechanics while providing context for specific functions.</p><h4>Setting up Dependencies</h4><p>We start by installing all of the required packages into the base environment. You’re welcome to create your conda environment, but this is a quick and easy way to start.</p><pre>import sys<br>import os<br>!{sys.executable} -m pip install langchain==0.0.335 --no-warn-script-location &gt; /dev/null<br>!{sys.executable} -m pip install pygpt4all==1.1.0 --no-warn-script-location &gt; /dev/null<br>!{sys.executable} -m pip install gpt4all==1.0.12 --no-warn-script-location &gt; /dev/null<br>!{sys.executable} -m pip install transformers==4.35.1 --no-warn-script-location &gt; /dev/null<br>!{sys.executable} -m pip install datasets==2.14.6 --no-warn-script-location &gt; /dev/null<br>!{sys.executable} -m pip install tiktoken==0.4.0 --no-warn-script-location &gt; /dev/null<br>!{sys.executable} -m pip install chromadb==0.4.15 --no-warn-script-location &gt; /dev/null<br>!{sys.executable} -m pip install sentence_transformers==2.2.2 --no-warn-script-location &gt; /dev/null</pre><p>These commands will install all the necessary packages into your base environment.</p><h4>The Data and Model</h4><p>We will be using a quantized version of Falcon 7B (gpt4all-falcon-q4_0) from the GPT4All project. You can learn more about this model on the <a href="https://gpt4all.io/index.html">GPT4ALL page</a> in the “Model Explorer” section. The model has been stored on disk to simplify the model access process.</p><p>The following logic downloads the available datasets from a Hugging Face project called <a href="https://huggingface.co/FunDialogues">FunDialogues</a>. The selected data will be passed through an embedding model and placed in our vector database in a subsequent step.</p><pre>def download_dataset(self, dataset):<br>        &quot;&quot;&quot;<br>        Downloads the specified dataset and saves it to the data path.<br><br>        Parameters<br>        ----------<br>        dataset : str<br>            The name of the dataset to be downloaded.<br>        &quot;&quot;&quot;<br>        self.data_path = dataset + &#39;_dialogues.txt&#39;<br><br>        if not os.path.isfile(self.data_path):<br><br>            datasets = {&quot;robot maintenance&quot;: &quot;FunDialogues/customer-service-robot-support&quot;, <br>                        &quot;basketball coach&quot;: &quot;FunDialogues/sports-basketball-coach&quot;, <br>                        &quot;physics professor&quot;: &quot;FunDialogues/academia-physics-office-hours&quot;,<br>                        &quot;grocery cashier&quot; : &quot;FunDialogues/customer-service-grocery-cashier&quot;}<br>            <br>            # Download the dialogue from hugging face<br>            dataset = load_dataset(f&quot;{datasets[dataset]}&quot;)<br>            # Convert the dataset to a pandas dataframe<br>            dialogues = dataset[&#39;train&#39;]<br>            df = pd.DataFrame(dialogues, columns=[&#39;id&#39;, &#39;description&#39;, &#39;dialogue&#39;])<br>            # Print the first 5 rows of the dataframe<br>            df.head()<br>            # only keep the dialogue column<br>            dialog_df = df[&#39;dialogue&#39;]<br>            <br>            # save the data to txt file<br>            dialog_df.to_csv(self.data_path, sep=&#39; &#39;, index=False)<br>        else:<br>            print(&#39;data already exists in path.&#39;)</pre><p>In the code snippet above, you can select from 4 different synthetic datasets:</p><ul><li><strong>Robot Maintenance</strong>: conversations between a technician and a customer support agent while troubleshooting a robot arm.</li><li><strong>Basketball Coach</strong>: conversations between basketball coaches and players during a game.</li><li><strong>Physics Professor</strong>: conversations between students and physics professor during office hours.</li><li><strong>Grocery Cashier</strong>: conversations between a grocery store cashier and customers</li></ul><h4>Configuring the Model</h4><p>The GPT4ALL extension in the LangChain API takes care of loading the model into memory and establishing a variety of parameters, such as:</p><ul><li><strong>model_path</strong>: This line specifies the file path for a pre-trained model.</li><li><strong>n_threads</strong>: Sets the number of threads to be used, which might influence parallel processing or inference speed. This is especially relevant for multi-core systems.</li><li><strong>max_tokens</strong>: Limits the number of tokens (words or subwords) for the input or output sequences, ensuring that the data fed into or produced by the model does not exceed this length.</li><li><strong>repeat_penalty</strong>: This parameter possibly penalizes repetitive content in the model’s output. A value greater than 1.0 prevents the model from generating repeated sequences.</li><li><strong>n_batch</strong>: Specifies the batch size for processing data. This can help optimize processing speed and memory usage.</li><li><strong>top_k</strong>: Defines the “top-k” sampling strategy during the model’s generation. When generating text, the model will consider only the top k most probable next tokens.</li></ul><pre>def load_model(self, n_threads, max_tokens, repeat_penalty, n_batch, top_k, temp):<br>        &quot;&quot;&quot;<br>        Loads the model with specified parameters for parallel processing.<br><br>        Parameters<br>        ----------<br>        n_threads : int<br>            The number of threads for parallel processing.<br>        max_tokens : int<br>            The maximum number of tokens for model prediction.<br>        repeat_penalty : float<br>            The penalty for repeated tokens in generation.<br>        n_batch : int<br>            The number of batches for processing.<br>        top_k : int<br>            The number of top k tokens to be considered in sampling.<br>        &quot;&quot;&quot;<br>        # Callbacks support token-wise streaming<br>        callbacks = [StreamingStdOutCallbackHandler()]<br>        # Verbose is required to pass to the callback manager<br><br>        self.llm = GPT4All(model=self.model_path, callbacks=callbacks, verbose=False,<br>                           n_threads=n_threads, n_predict=max_tokens, repeat_penalty=repeat_penalty, <br>                           n_batch=n_batch, top_k=top_k, temp=temp)</pre><h4>Building the Vector Database with ChromaDB</h4><p>The Chroma vector database is an integral part of our RAG setup, where we store and manage our data efficiently. Here’s how we build it:</p><pre>def build_vectordb(self, chunk_size, overlap):<br>        &quot;&quot;&quot;<br>        Builds a vector database from the dataset for retrieval purposes.<br><br>        Parameters<br>        ----------<br>        chunk_size : int<br>            The size of text chunks for vectorization.<br>        overlap : int<br>            The overlap size between chunks.<br>        &quot;&quot;&quot;<br>        loader = TextLoader(self.data_path)<br>        # Text Splitter<br>        text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=overlap)<br>        # Embed the document and store into chroma DB<br>        self.index = VectorstoreIndexCreator(embedding= HuggingFaceEmbeddings(), text_splitter=text_splitter).from_loaders([loader])</pre><h4>Executing the Retrieval Mechanism</h4><p>Upon receiving a user’s query, we use similarity search to search our vector DB for similar data. Once a k number of matching results are found, they are retrieved and used to add the context to the user’s query. We use the PromptTemplate function to build a template and embed the user’s query alongside the retrieved context. Once the template has been populated, we move on to the inference component.</p><pre>def retrieval_mechanism(self, user_input, top_k=1, context_verbosity = False, rag_off= False):<br>        &quot;&quot;&quot;<br>        Retrieves relevant document snippets based on the user&#39;s query.<br><br>        Parameters<br>        ----------<br>        user_input : str<br>            The user&#39;s input or query.<br>        top_k : int, optional<br>            The number of top results to return, by default 1.<br>        context_verbosity : bool, optional<br>            If True, additional context information is printed, by default False.<br>        rag_off : bool, optional<br>            If True, disables the retrieval-augmented generation, by default False.<br>        &quot;&quot;&quot;<br><br>        self.user_input = user_input<br>        self.context_verbosity = context_verbosity<br>                <br>        # perform a similarity search and retrieve the context from our documents<br>        results = self.index.vectorstore.similarity_search(self.user_input, k=top_k)<br>        # join all context information into one string <br>        context = &quot;\n&quot;.join([document.page_content for document in results])<br>        if self.context_verbosity:<br>            print(f&quot;Retrieving information related to your question...&quot;)<br>            print(f&quot;Found this content which is most similar to your question: {context}&quot;)<br><br>        if rag_off:<br>            template = &quot;&quot;&quot;Question: {question}<br>            Answer: This is the response: &quot;&quot;&quot;<br>            self.prompt = PromptTemplate(template=template, input_variables=[&quot;question&quot;])<br>        else:     <br>            template = &quot;&quot;&quot; Don&#39;t just repeat the following context, use it in combination with your knowledge to improve your answer to the question:{context}<br><br>            Question: {question}<br>            &quot;&quot;&quot;<br>            self.prompt = PromptTemplate(template=template, input_variables=[&quot;context&quot;, &quot;question&quot;]).partial(context=context)</pre><p>The LangChain LLMChain utility to execute inference based on the query passed by the user and the configured template. The result is returned to the user.</p><pre> def inference(self):<br>        &quot;&quot;&quot;<br>        Performs inference to generate a response based on the user&#39;s query.<br><br>        Returns<br>        -------<br>        str<br>            The generated response.<br>        &quot;&quot;&quot;<br><br>        if self.context_verbosity:<br>            print(f&quot;Your Query: {self.prompt}&quot;)<br>            <br>        llm_chain = LLMChain(prompt=self.prompt, llm=self.llm)<br>        print(&quot;Processing the information with gpt4all...\n&quot;)<br>        response = llm_chain.run(self.user_input)<br><br>        return  response  </pre><h4>Interactive Experimentation</h4><p>To help you get started quickly, the notebook includes integrated ipywidget components. You must run all the cells in the notebook to enable these components. We encourage you to adjust the parameters and evaluate the impact on the latency and fidelity of the system’s response. Remember, this is just a starting point and a basic demonstration of RAG’s capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vojVfoUKH41VOAs18BQA0Q.png" /><figcaption>Figure 10. In this example, we get a quick taste of the power of RAG, clearly seeing the benefits of the additional context provided by RAG, which helps yield a helpful answer to the user’s question — “My robot is not turning on, Can you help me?” The RAG-enabled output provides valid recommendations, while the raw model without RAG simply provides a polite inquiry that is not very helpful to the user.</figcaption></figure><h3>Summary and Discussion</h3><p>No one wants to interact with slow, unstable chatbots that respond with bogus information. There are a plethora of technical stack combinations to help developers avoid building systems that yield terrible user experiences. In this article, we have interpreted the importance of inference engine quality to the user experience from the perspective of a stack that enables scale, fidelity, and latency benefits. The combination of RAG, CPUs, and model optimization techniques checks all corners of the IQ Triangle (Figure 3), aligning well with the needs of operational LLM-based AI chat applications.</p><p><strong>A few exciting things to try would be:</strong></p><ul><li>Edit the prompt template found in the retrieval_mechanism method to engineer better prompts in tandem with the retrieved context.</li><li>Adjust the various model and RAG-specific parameters and evaluate the impact on inference latency and response quality.</li><li>Add new datasets that are meaningful to your domain and test the viability of using RAG to build your AI chat-based applications.</li><li>This example’s model (gpt4all-falcon-q4_0) is not optimized for Xeon processors. Explore using models that are optimized for CPU platforms and evaluate the inference latency benefits.</li></ul><p><strong><em>Thank you for reading! Don’t forget to follow </em></strong><a href="https://eduand-alvarez.medium.com/"><strong><em>my profile for more articles</em></strong></a><strong><em> like this!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5d55f398502" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/retrieval-augmented-generation-rag-inference-engines-with-langchain-on-cpus-d5d55f398502">Retrieval Augmented Generation (RAG) Inference Engines with LangChain on CPUs</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>