<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rachel Shalom on Medium]]></title>
        <description><![CDATA[Stories by Rachel Shalom on Medium]]></description>
        <link>https://medium.com/@rshalom1984?source=rss-471c0ed81e43------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*I6y25qbFbMsaZFsR8YoWOw.jpeg</url>
            <title>Stories by Rachel Shalom on Medium</title>
            <link>https://medium.com/@rshalom1984?source=rss-471c0ed81e43------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:21:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rshalom1984/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Can you really AutoReseach?]]></title>
            <link>https://medium.com/@rshalom1984/can-you-really-autoreseach-f5354a7653ff?source=rss-471c0ed81e43------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5354a7653ff</guid>
            <category><![CDATA[autonomous-research]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Rachel Shalom]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 17:13:21 GMT</pubDate>
            <atom:updated>2026-04-01T17:13:21.606Z</atom:updated>
            <content:encoded><![CDATA[<p>A few weeks ago, Andrej Karpathy released an<a href="https://github.com/karpathy/autoresearch"> AutoResearch GitHub repo</a> showing how he used an agent to train a mini GPT on a single H100, including the training data and the resulting improvements. You can find more details <a href="https://x.com/karpathy/status/2029701092347630069">here </a>and in this <a href="https://x.com/karpathy/status/2029701092347630069">tweet</a>.</p><p>I decided to try something similar. At Dell, we had a use case where part of my experiments involved fine-tuning <a href="https://huggingface.co/nomic-ai/nomic-embed-text-v1.5">nomic-embed-text-v1 model</a>. This is a 137M parameter model with 8192 context length that was trained using the BERT architecture with some modifications.<br>Since I needed to show results quickly, AutoResearch seemed like a good fit. The goal is to optimize the model for better document retrieval — i.e., when a user asks a question, we retrieve the most relevant docs stored as <br>embeddings.</p><h4>Experiment setup</h4><p>I already had the training data prepared and a training loop in place. I used a similar <a href="https://github.com/nomic-ai/contrastors">training method </a>to nomic-ai with medications. I also <br>modified program.md to point to my train.py, and used Claude Opus 4.6, instructing it to follow the MD file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LVlYO09Vu8R3IDl8x6H0vQ.png" /></figure><p>The agent is given a training script and a config file that it can freely modify, with a strict experiment loop: make a change, commit it, run<br> a short (~10-minute) training run, and check whether the loss improved.<br> If loss improves, keep the commit and build on it; if not, revert and try something else. Training data is randomized each run so results aren’t tied to a fixed subset, and the single optimization target is loss, accuracy is tracked only as a sanity check.<br>The agent must never stop to ask for permission. It runs autonomously and indefinitely until manually interrupted, so it can run overnight and<br> produce dozens of experiments. It shouldn’t just follow a checklist, it should act like a researcher: form hypotheses based on observations and propose its own experiments.</p><p>The agent could only modify train.py and train_config.py (i.e., training parameters and logic).<br> Hardware used: a Dell server with 8× H100 80GB GPUs and 3 TB memory.</p><p>I let it run from noon until the next morning. It completed 135 experiments and saved the results to a TSV file.</p><h4>Context on contrastive training for embedding models</h4><p>Contrastive fine-tuning teaches a model to place similar pieces of text closer together and different text farther apart in its internal “map” of meaning. For a query, it learns to pull the correct answer closer (positive example) and push similar-but-incorrect text away (negative examples). This improves the model’s ability to capture nuance and retrieve the most relevant results.</p><p>I used <strong>InfoNCE contrastive loss</strong>, which is commonly applied in embedding models and RAG systems to refine semantic similarity. To train the model, I structured the data as <strong>triplets</strong>: a query, one relevant (positive) document, and multiple non-relevant but potentially similar (negative) documents. In my setup, each example includes <strong>one positive and seven negative documents</strong>. Increasing the number of negatives typically improves performance by providing more “distractors,” forcing the model to learn sharper, more discriminative embeddings.</p><p>The training objective is to minimize the distance between the query and the positive document while maximizing the distance from the negatives. As a result, the model learns embedding vectors where related queries and documents are close together in the embedding space, while irrelevant ones are pushed farther apart.</p><p><strong>Examples:</strong></p><blockquote>Query: <em>“How do I open a GitHub account?”</em></blockquote><blockquote>Positive: <em>Step-by-step guide to creating a GitHub account</em></blockquote><blockquote>Negatives: <em>How to join a Slack workspace</em>, <em>Setting up a GitLab repository</em>, <em>Managing GitHub permissions</em></blockquote><blockquote>Query: <em>“How to reset my password?”</em></blockquote><blockquote>Positive: <em>Instructions to reset your account password</em></blockquote><blockquote>Negatives: <em>How to update your email address</em>, <em>Account security best practices</em>, <em>Troubleshooting login issues</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xeVS30OZhRl4wsmfkVZTWg.png" /></figure><h4>Results and analysis of the process</h4><p>Out of 135 experiments the agent made progress with 15 experiments and discarded the rest</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IfMPetlDi6OBfvjGq8d9ZQ.png" /></figure><h3>Analysis of the agent experiments</h3><h4>Core takeaway</h4><p>The agent was effective at optimizing hyperparameters and improving training metrics.<br> However, some improvements came from <strong>changing the difficulty or structure of the task</strong>, rather than improving the model’s ability to learn meaningful semantic relationships.</p><pre>commit loss accuracy throughput status description<br>d589193 12.4579 0.4961 7.0 keep baseline: bs=256, lr=2e-5, warmup=10, linear, doc_len=512, 4GPU<br>9714f56 9.7395 0.5977 7.0 keep exp1: lr=1e-4 (5x baseline)<br>a8c0ee0 10.0924 0.4961 7.0 discard exp2: cosine schedule (worse loss)<br>f74d70e 9.7234 0.5156 7.0 discard exp3: lr=2e-4 (no improvement, acc dropped)<br>e1ec3d2 9.9420 0.5391 7.0 discard exp4: weight_decay=0.001 (worse loss &amp; acc)<br>08973bc 9.2806 0.5781 7.0 keep exp5: grad_accum=2 (effective bs=512, loss improved)<br>- 9.2989 0.5234 7.0 discard exp6: grad_accum=4 (no improvement, fewer optimizer steps)<br>- 12.1559 0.5313 7.0 discard exp7: logit_scale_init=14.0 (much worse loss)<br>- 9.8683 0.5781 7.0 discard exp8: max_grad_norm=5.0 (worse loss)<br>- - - - crash exp9: num_negatives=15 (too slow, ~18s/step, killed)<br>- 10.7103 0.4961 7.5 discard exp10: warmup=5, grad_cache_chunk=64 (worse loss, higher VRAM)<br>- 13.4701 0.5430 7.0 discard exp11: label_smoothing=0.1 (inflated loss, acc worse)<br>- 27.6029 0.4063 7.0 discard exp12: 8 GPUs (loss scaled by world_size=8, much worse)<br>81a2da5 7.8291 0.5547 14.0 keep exp13: bs=128, grad_accum=1 (2x optimizer steps, loss dropped)<br>- 8.0165 0.5938 28.0 discard exp14: bs=64 (slightly worse loss despite more steps)<br>- 8.2791 0.5625 14.0 discard exp15: 160 steps (stalled at 121, ran out of data)<br>- 9.3260 0.6094 14.0 discard exp16: lr=5e-5 with bs=128 (too slow convergence)<br>......<br>......<br>......</pre><h4>What worked</h4><p><strong>Learning rate tuning (how fast the model learns)</strong><br> This had the biggest impact.</p><ul><li>Too low → learning is slow (the model underfits)</li><li>Too high → training becomes unstable (the model overshoots good solutions)</li><li>Best range: ~1.5e-4 to 1.75e-4<br> This is expected — learning rate is usually the most influential parameter.</li></ul><p><strong>Gradient clipping (max_grad_norm ≈ 1.5)</strong><br> Gradient clipping limits how large each model update step can be.<br> Intuition: if the model tries to “jump too far” in one step, we cap it to keep training stable. This helped prevent instability and improved convergence.</p><p><strong>Smaller batch sizes (e.g., bs=32–128)</strong><br> Smaller batches mean the model sees fewer examples per step and updates more frequently. This can help optimization (more frequent updates), but it also makes the loss noisier.<br> Important: lower loss with smaller batches doesn’t necessarily mean a better model — it can simply reflect different training dynamics.</p><h4>What didn’t help (Optional reading, you can skip this part if you dont want to dive in)</h4><p><strong>Schedulers :</strong>A scheduler controls how the learning rate changes during training. It’s like deciding whether to take big steps early and smaller steps later during training<br><em>What happened here:</em> Simple <strong>linear decay</strong> worked best — gradually reducing the step size was enough. More complex schedules (like cosine) didn’t add value and sometimes slowed convergence or led to worse results.</p><p><strong>Regularization (weight decay, label smoothing)</strong><br> <em>What it is:</em> Techniques to prevent overfitting (i.e., the model memorizing instead of generalizing).<br> <em>Intuition:</em></p><ul><li><strong>Weight decay</strong> slightly penalizes large weights → keeps the model “simpler”</li><li><strong>Label smoothing</strong> softens targets → prevents overconfidence<br> <em>What happened here:</em> The model likely wasn’t overfitting, so adding regularization just made learning harder and hurt performance.</li></ul><p><strong>.Optimizer tweaks (beta1, beta2, eps)</strong><br> <em>What it is:</em> Internal parameters of optimizers like Adam that control how gradients are averaged and stabilized.<br> <em>Intuition:</em> These settings affect how “smooth” or “responsive” updates are over time.<br> <em>What happened here:</em> The default values are already well-tuned for most cases, so changing them didn’t meaningfully improve trainiing.</p><h4>Important nuance: better metrics don’t always mean a better model</h4><p>Some changes improved loss/accuracy but weakened the actual goal of contrastive learning:</p><p><strong>Reducing or removing negatives (e.g., 0–3 negatives) l</strong>eads to much lower loss and higher accuracy, but removes or weakens the contrastive signal.<br> 👉 The model is no longer learning to distinguish relevant vs. irrelevant text as effectively<br> 👉 This means “better metrics, but not necessarily better retrieval quality”</p><p><strong>Reducing document length </strong>Makes the task easier (less information to process) but hurts performance for real use cases with long documents</p><p>These are not “wrong” experiments — but they highlight that <strong>training metrics alone can be misleading</strong>.</p><h3>My thoughts conclusions</h3><p>Working with an LLM agent/harness feels like collaborating with a very smart postdoc: fast, creative, and productive, while at the same time occasionally like guiding a 10-year-old who misses key context.</p><p><strong>Did it save me time? Yes.</strong><br> It explored a large space of experiments quickly and surfaced useful directions. But it still required close supervision — reviewing results, correcting course, and filtering out changes that didn’t make sense for the use case. Some of its “optimizations” improved metrics but broke the logic of the task.<br> For example:</p><ul><li>Reducing context length from 2048 to 512 lowered the loss — but doesn’t fit a long-context retrieval use case</li><li>Removing negatives entirely improved metrics — but defeats the purpose of contrastive training</li></ul><p>So while the numbers looked better, the model itself wasn’t aligned with the goal.</p><p><strong>Is this “auto-research”? Not quite.</strong><br> It’s not replacing reasoning or domain understanding. But it is a strong <strong>accelerator</strong> — it expands how fast you can iterate and explore.</p><p>The bigger shift is not just in deep learning — it’s a new way of working:<br> Define a system, define a measurable objective, and let the agent optimize against it.</p><h3>Where this becomes powerful</h3><p>This approach is powerful in <strong>any field where success can be measured clearly</strong>. Once you can define an objective and evaluate the outcome, an agent can start exploring options and optimizing toward that goal.</p><p>For example:</p><ul><li><strong>Kernel or systems optimization</strong>: this is a great fit because evaluation is straightforward — runtime, memory use, throughput, latency. If a change makes the process run faster or more efficiently, the signal is clear.</li><li><strong>Model training</strong>: loss, accuracy, Recall@K, or other retrieval metrics can be used as the optimization target.</li><li><strong>Presentation design</strong>: even something that feels subjective can become optimizable if you define a scoring function. For example:<br> +1 for using a blue color scheme<br> +1 for using internal terminology correctly<br> +1 for including required sections<br> +1 for keeping slides under a certain word count<br> Then the agent can optimize against that score.</li></ul><p>The interesting shift is that this is not limited to “AI research” or even to engineering. It can apply anywhere you can turn taste, quality, speed, or usefulness into a measurable signal. The challenge is no longer only building the system — it is defining the right evaluation. Once that exists, a whole new class of optimization problems becomes accessible.<br><strong>The real unlock is not the agent itself — it’s the ability to define what “good” looks like in a way the agent can optimize.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5354a7653ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MXFP4 Quantization and GPT-OSS]]></title>
            <link>https://medium.com/@rshalom1984/mxfp4-quantization-and-gpt-oss-119383e8d4c5?source=rss-471c0ed81e43------2</link>
            <guid isPermaLink="false">https://medium.com/p/119383e8d4c5</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[reasoning]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[gpt-oss]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Rachel Shalom]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 11:52:15 GMT</pubDate>
            <atom:updated>2025-09-17T11:52:15.626Z</atom:updated>
            <content:encoded><![CDATA[<p>GPT-OSS dropped in early August, and it’s been the talk of the town. It’s a big deal because it’s the first open model we’ve seen from Open AI since their GPT-2 architecture came out more than five years ago.</p><p>I’ve been going through their technical report, and while there’s a lot to unpack, I’m thinking of breaking this post down into two parts:</p><p>1. A quick look at the report’s main takeaways and what’s interesting at a high level.</p><p>2. Deeper dive into the most interesting part, in my opinion: the post-training and quantization with MXFP4</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ejkrhfNoTJGGJBaEG8RrSw.png" /></figure><h3><strong>Overview and main takeaways:</strong></h3><h4><strong>Post training: A focus on reasoning and Tool Use</strong></h4><p>The model undergo post training using chain of thought reinforcement learning (CoT RL) a technique similar to what’s used in OpenAI’s o3 models.</p><p>Here&#39;s a quick rundown of what that means:</p><ul><li><strong>Chain of Thought (CoT)</strong> is basically the model “showing its work” — it breaks down tough problems step by step instead of just giving a final answer.</li><li><strong>CoT RL:</strong> The main assumption behind this approach is that explicitly training a model to reason through problems using CoT is far more effective than just having it implicitly learn patterns from labeled data. By using reinforcement learning, the model is rewarded for generating accurate and well-reasoned “chains of thought,” pushing it to develop more robust and explicit problem-solving skills.</li></ul><p>This technique isn’t new. You can find an excellent, explained example of a similar <strong>CoT-RL</strong> process from DeepSeek right <a href="https://github.com/deepseek-ai/DeepSeek-R1">here</a>.</p><h4><strong>Tool calls</strong></h4><p>GPT-OSS’s training also focused on <strong>tool calls</strong>, teaching it to use external functions and APIs. Here are the three main types of tool calls the model was trained on:</p><ul><li><strong>Web Browsing:</strong> The model was trained to interact with a search tool by doing three things: generating search queries, navigating to specific web pages, and searching for content within a page.</li><li><strong>Python Execution:</strong> It can execute code in a stateful Jupyter notebook, allowing it to perform calculations and data manipulation.</li><li><strong>Arbitrary Function Calls:</strong> The model can call any function described in the prompt using a defined schema, similar to how a developer would work with an API.</li></ul><p>If your application relies heavily on search and tool calls, this model is a great one to consider.</p><h4><strong>Reasoning Levels</strong></h4><p>OpenAI also introduced different levels of reasoning: <strong>low, medium, and high</strong>. By simply adding Reasoning: low to a prompt, you can control the model&#39;s output. Increasing the reasoning level causes the model&#39;s average response length to grow, as it generates a more tokens, or “thinking longer”.</p><h4>Architecture and models size:</h4><p>The GPT-OSS models are built on a foundation of autoregressive Mixture-of-Experts (MoE) transformers.<br>The models come in two primary sizes:</p><ul><li>GPT-OSS-120b: with 36 layers and a massive 116.8 billion total parameters. Where 5.1 billion of these are active for each token during a forward pass.</li><li>GPT-OSS-20b: with 24 layers, and 20.9 billion total parameters where 3.6 billion active parameters per token.</li></ul><h4><strong>Harmony chat format: A new Standard</strong></h4><p>Many models are using xml like special tokens: like qwen and mistral models it is used for beginning and end of message, role based ( user, system and assistant) thinking/reasoning and tool calls.</p><p>They opened source a completely new design of response:</p><ul><li>Multi-channel architecture: Messages can be tagged as <em>analysis </em>(reasoning), <em>commentary </em>(tool preambles), or <em>final </em>(user-facing)</li><li>Role hierarchy: system &gt; developer &gt; user &gt; assistant &gt; tool (for handling instruction conflicts)</li><li>Message routing: Uses to syntax for directing messages between components</li><li>TypeScript-style tool definitions: More expressive and concise than JSON schemas</li><li>Separate safety standards per channel: The analysis channel isn’t safety-filtered like final</li></ul><p>Here is an example of user input and a raw model output with Harmony:</p><pre>What is 2+2?</pre><p>Model output (Harmony format)</p><pre>&lt;|channel|&gt;analysis&lt;|message|&gt;User asks simple addition. Answer directly.&lt;|end|&gt;<br>&lt;|start|&gt;assistant&lt;|channel|&gt;final&lt;|message|&gt;The result is 4.&lt;|return|&gt;</pre><p><strong>Evaluation and Benchmarks</strong>:</p><p>The GPT-OSS models were put through a rigorous evaluation process that tested their core capabilities, tool use, multilingual performance, and safety. However, the evaluation results were only compared against OpenAI’s own models (o3, o3-mini, and o4-mini).</p><p>This decision to omit comparisons with other leading open-source or commercial models (like Mistral, DeepSeek, Qwen, or Claude) is a notable exclusion. It’s a bit odd not to see a direct benchmark against the broader ecosystem.</p><p>Naturally, the community has already stepped in to fill this gap. Early comparisons are beginning to emerge, with some papers already showing interesting results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*1p38ntB2mRpjVzS_fJkkzQ.png" /></figure><p><a href="https://www.google.com/search?q=insert_link_to_paper_here"><strong>You can read more details in this paper</strong></a>.</p><h3>Quantization and MXFP4</h3><p>Before we get to the specifics of MXFP4, let’s start with the basics of quantization.</p><p>So, why is it so crucial for large models? It all comes down to their massive size and the huge VRAM requirements that come with it.</p><p>To put this into perspective, a 120B parameter model needs an incredible amount of memory:</p><ul><li>In <strong>FP32</strong> (4 bytes per parameter), it would require <strong>480 GB</strong> of VRAM.</li><li>Even in <strong>FP16</strong> (2 bytes per parameter), that’s still <strong>240 GB</strong>.</li></ul><p>That’s a lot, and it’s simply not feasible for most hardware. Quantization is the solution, reducing the precision of a model to shrink its size.</p><p>So what type of quantization did OpenAI use for GPT-OSS?</p><h3>Enter MXFP4</h3><p>GPT-OSS uses <strong>MXFP4</strong> — or <strong>Microscaling FP4</strong> — a next-generation, 4-bit floating-point format standardized in early 2024 by the Open Compute Project (OCP), with support from tech giants like NVIDIA, AMD, Microsoft, and OpenAI.</p><p>While quantization isn’t a new concept, it’s often seen as a compromise, as lowering precision can lead to a loss in model quality. MXFP4 offers significantly better dynamic range and model accuracy compared to other traditional quantization( like INT4 or FP4). It achieves this by combining a low-precision 4-bit floating-point format with a higher-precision, block-specific scaling factor.</p><h3>How Does it Work?</h3><p>To understand MXFP4, you first need to understand its components. The FP4 format is a 4-bit floating-point number that consists of three parts:</p><ul><li><strong>1 sign bit (S):</strong> This single bit tells you if the number is positive or negative.</li><li><strong>2 exponent bits (E):</strong> These bits determine the number’s scale or magnitude.</li><li><strong>1 mantissa bit (M):</strong> This represents the number’s fractional part, which defines its precision.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*NVoV_6FvZ0jeSt7QkzkADw.png" /></figure><p>This E2M1 structure allows for a specific set of 15 quantized values: [0, ±0.5, ±1.0, ±1.5, ±2.0, ±3.0, ±4.0, ±6.0]. The values are determined by the combination of the sign, exponent, and mantissa bits, which define the range and precision of the format.</p><h4>The Deep Dive: How MXFP4 Works</h4><p>Now that we’ve covered the basics of quantization, let’s dive into what makes <strong>MXFP4</strong> so effective. Instead of quantizing each value in a model’s tensor independently, MXFP4 uses a technique called <strong>microscaling</strong>. It divides the model’s data into small blocks, typically 32 elements at a time. All 32 of these values then share a single, common scaling factor.</p><p>This approach lets a 4-bit format represent a much wider range of numbers than it could on its own, preserving the model’s quality with minimal loss.</p><p><strong>Here’s a step-by-step breakdown of the process:</strong></p><p><strong>Step 1: Block Formation</strong></p><p>Think of the model’s large tensor of numbers. MXFP4 doesn’t look at all the numbers in the tensor. Instead, it groups them into small blocks of 32 numbers each.</p><p><strong>Step 2: Determining the Scale</strong></p><p>The system finds the number with the largest absolute value in that block. This largest value is then used to calculate a single, shared E8M0 scale factor for the entire block. This scale factor is essentially an 8-bit integer that represents a power of two. For example, if the largest number in a block is 12,288, the system finds the smallest power of two that can “scale” this number into the FP4 range. In this case, that number is 2048 (2^11). This value becomes the E8M0 scale factor for the entire block.</p><p><strong>Step 3: Quantization<br></strong>With the scale factor determined, every number in the block is divided by that factor and then quantized to the nearest available FP4 value. Our largest number, 12,288, is divided by 2048 to become 6.0, which is a valid FP4 value. Another number, say 2560, becomes approximately 1.25, which is then rounded to the nearest FP4 value (1.5). The system stores the 32 new 4-bit values along with the single 8-bit scale factor for the block.</p><p><strong>Step 4: De-quantization<br></strong>When the model needs to perform a calculation, it retrieves the stored 4-bit values and multiplies them back by the shared E8M0 scale factor. The 4-bit value of 6.0 becomes 6.0 x 2048 = 12,288. The 4-bit value of 1.5 becomes 1.5 x 2048 = 3,072. It’s important to remember that this is a lossy process, as the restored value (3,072) is based on the rounded number (1.5) and not the original value (2,560). However, the MXFP4 method is designed to minimize this loss, preserving the model’s performance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P227EqwcSdZncVllQW9fdQ.png" /></figure><h4>Inference and Training</h4><p>Let’s look at how MXFP4 is put into practice for both inference and training.</p><h4><strong>How Inference Works</strong></h4><p>For inference, the process is very efficient. The quantized model, with its weights in the compact 4-bit MXFP4 format, is loaded into the GPU’s memory, taking up a fraction of the VRAM. When a calculation is needed, the following happens:</p><ul><li><strong>On-the-Fly Dequantization:</strong> The 4-bit weights are read from memory. Just before the computation, specialized hardware (like NVIDIA’s Tensor Cores) quickly multiplies them by their 8-bit scale factor to restore their original magnitude. This process is extremely fast and happens in real-time.</li><li><strong>Fast Computation:</strong> The restored values in higher precision format, are then used for calculations like matrix multiplications.</li></ul><p>The key is that the data is only de-quantized for the computation and not for storage, which saves a significant amount of memory.</p><h4>How Training Works</h4><p>GPT-OSS was post trained with quantization-aware fine-tuning (QAT). During QAT, the weights and/or activations of the model are ”fake quantized” in 2 steps:</p><ol><li><strong>Quantize</strong>: It takes the full-precision floating-point value (e.g., BF16) and rounds it to the nearest value representable by the target lower-precision format (e.g., 4-bit integer).</li><li><strong>Dequantize</strong>: That rounded value is immediately converted back to its original full-precision format. The result is a floating-point number that perfectly mimics the limited precision of the quantized format. This “fake” quantization happens during the forward pass of the training process, so the model learns with the constraints of the final quantized format.</li></ol><p>3.<strong> Full precision backward pass:</strong> the backward pass — where gradients are calculated and model parameters are updated — is still performed in full precision.</p><p><strong>This allows the model to learn to adjust its parameters to minimize both the <em>prediction error </em>and the <em>quantization error</em>.</strong> By experiencing quantization errors during training, the model learns to adjust its parameters to mitigate the negative effects of reduced precision . This adaptation makes the model more robust to quantization, leading to better accuracy when the model is actually quantized for deployment.</p><p>If you want to know more I recommend this <a href="https://arxiv.org/abs/2411.11038">paper</a></p><p><strong>Let’s look at some basic code:</strong></p><pre>from transformers import AutoModelForCausalLM, AutoTokenizer<br>model_id = &quot;openai/gpt-oss-20b&quot;<br>tokenizer = AutoTokenizer.from_pretrained(model_id)<br>model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=&quot;auto&quot;,device_map=&quot;cuda&quot;)<br>messages = [<br>    {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;Always respond in riddles&quot;},<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is the weather like in Madrid?&quot;},<br>]<br><br>inputs = tokenizer.apply_chat_template(<br>    messages,<br>    add_generation_prompt=True,<br>    return_tensors=&quot;pt&quot;,<br>    return_dict=True,<br>    resaoning_effort=&quot;high&quot;<br>).to(model.device)<br><br>generated = model.generate(**inputs, max_new_tokens=500)<br>print(tokenizer.decode(generated[0][inputs[&quot;input_ids&quot;].shape[-1]:]))</pre><p>Here is the the code in <a href="https://colab.research.google.com/drive/1AtrSZJmbGodTznqRxq6JiEk1evLSKeiy?usp=sharing">google-colab</a> if you want to run it.</p><h4>The Hardware Ecosystem: Who Supports MXFP4?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f3ZFskKksGiC3l88e3LTpA.png" /><figcaption>source: <a href="https://www.gigxp.com/the-mxfp4-revolution/">https://www.gigxp.com/the-mxfp4-revolution/</a></figcaption></figure><h4>Summary and Final Thoughts</h4><p>While GPT-OSS is packed with innovation — from the Harmony format to its quantization — it’s worth noting that its core methodologies aren’t entirely new. The real test will be to see how the industry reacts and adopts this model for real-world use cases. I personally enjoyed diving deep into the details of quantization, and I hope you did too.</p><p><strong>Resources:</strong></p><p><a href="https://huggingface.co/blog/kuotient/chatml-vs-harmony">https://huggingface.co/blog/kuotient/chatml-vs-harmony</a></p><p><a href="https://github.com/openai/gpt-oss/tree/main?tab=readme-ov-file#tools">https://github.com/openai/gpt-oss/tree/main?tab=readme-ov-file#tools</a></p><p><a href="https://apxml.com/courses/mixture-of-experts-advanced-implementation/chapter-4-efficient-moe-inference/moe-quantization-techniques">https://apxml.com/courses/mixture-of-experts-advanced-implementation/chapter-4-efficient-moe-inference/moe-quantization-techniques</a></p><p><a href="https://arxiv.org/abs/2508.12461?utm_source=chatgpt.com">Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI’s Latest Open Source Models</a></p><p><a href="https://www.gigxp.com/the-mxfp4-revolution/#google_vignette">https://www.gigxp.com/the-mxfp4-revolution/#google_vignette</a></p><p><a href="https://buzzgrewal.medium.com/mxfp4-fp4-and-fp8-how-gpt-oss-runs-120b-parameters-on-an-80gb-gpu-with-moe-weight-quantization-db26b57fd787">https://buzzgrewal.medium.com/mxfp4-fp4-and-fp8-how-gpt-oss-runs-120b-parameters-on-an-80gb-gpu-with-moe-weight-quantization-db26b57fd787</a></p><p><a href="https://developer.nvidia.com/blog/fine-tuning-gpt-oss-for-accuracy-and-performance-with-quantization-aware-training/">https://developer.nvidia.com/blog/fine-tuning-gpt-oss-for-accuracy-and-performance-with-quantization-aware-training/</a></p><p><a href="https://medium.com/better-ml/quantization-aware-training-qat-vs-post-training-quantization-ptq-cd3244f43d9a">https://medium.com/better-ml/quantization-aware-training-qat-vs-post-training-quantization-ptq-cd3244f43d9a</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=119383e8d4c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DevOps made easy with GenAI]]></title>
            <link>https://medium.com/@rshalom1984/devops-made-easy-with-genai-55b8ed25b3a8?source=rss-471c0ed81e43------2</link>
            <guid isPermaLink="false">https://medium.com/p/55b8ed25b3a8</guid>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[gen-ai-for-business]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Rachel Shalom]]></dc:creator>
            <pubDate>Fri, 19 Jan 2024 08:59:31 GMT</pubDate>
            <atom:updated>2024-01-19T08:59:31.953Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, I will tell you about a winning hackathon project at Dell global hackathon. The idea is to use <strong>Gen-AI</strong> to help <strong>lower the barrier to entry</strong> and simplify the creation of infrastructure as code templates. By leveraging Gen-AI, we aim to streamline the development workflow and enhance the overall efficiency of managing complex infrastructure.</p><h4>Infrastructure As Code</h4><p>Think about Infrastructure as Code (IaC) as having a magic book that tells computers how to build themselves, making things quicker, easier, and less error-prone for people who work with computer systems.</p><p><strong>DevOps automation</strong> plays a crucial role in simplifying the management of cloud infrastructure. By utilizing <strong>Infrastructure as Code</strong>, we transform infrastructure resources into software assets. This approach allows us to leverage software tools for automating tasks such as provisioning, configuration, and ongoing maintenance of these resources.</p><h4>The challenge with IaC</h4><p>It’s essential to acknowledge that writing IaC is still a <strong>challenging endeavor</strong>. It demands a rare combination of both <strong>development skills</strong> and <strong>operational expertise</strong>. Additionally, IaC involves interacting with live systems, which can be rather <strong>complex</strong> to debug and troubleshoot due to their distributed and asynchronous nature.</p><p>This complexity is amplified when dealing with <strong>edge infrastructure</strong>, which, by definition, is more distributed and heterogeneous. As a result, the development process becomes significantly more intricate.</p><h4>Using Gen-AI to reduce the barrier to entry of developing IaC</h4><p>One of the key benefits of using natural language to generate code is the ability to reduce significantly the learning curve that is often associated with code development in general and devops automation specifically.</p><p>Indeed according to a recent research by McKinsey — AI Unleash: Generative AI is poised to transform software development in a way that no other tooling or process improvement has done… As the technology evolves and is seamlessly integrated within tools across the software development life cycle, it is expected to further improve the speed and even quality of the development process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*wR_C7e2P2Cb7VhU0a8HZjA.png" /></figure><h4>The challenges with out of the box Gen-AI tools</h4><p>Handling custom development patterns: The challenges of using the out-box gen-ai tools such as GitHub co-pilot is that it was pre-trained on generic language and development patterns therefore requires special fine-tuning to address more special languages and development patterns.</p><p>To address those challenges lets start by examining a real life use case and follow the steps that a typical DevOps engineers of a software company need to go through to develop the IaC code for automating the provisioning and configuration of his software product.</p><p>We will then explore the steps that we took to solve this challenge of generating IaC templates that will be fit this specific use case.</p><h4>Exploring Edge Orchestration Through a Customer Scenario</h4><p>Let’s delve into the realm of edge orchestration with a practical example involving one of Dell’s clients. Picture a video analytics company that specializes in analyzing footage from cameras installed in retail environments, such as supermarkets and shopping centers. Their goal is to enhance the sales performance of their clients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*c_oYVROaEGs9wpNqeG0vSw.png" /></figure><p>This customer faced the challenge of deploying their software application across 5,000 cameras, serving as edge devices in various shopping centers. This complex task involved several steps like:</p><ol><li>Selecting the appropriate Operating System (OS) for installation.</li><li>Establishing a file system.</li><li>Implementing their software onto these 5,000 cameras.</li><li>Configuring the network to connect these cameras with their application.</li><li>Scheduling an automatic shutdown of the cameras between 10:00 PM to 6:00 AM.</li><li>Planning for periodic software updates.</li></ol><p>Each of these tasks represents a component of a configuration file. This file acts as a detailed blueprint, outlining the necessary environment and services to be set up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*HX_E9SDUf75jfiCcUosfbg.png" /></figure><p>For a demonstration tailored to this customer, Dell typically assigns a solution engineer to craft this blueprint.</p><p>This process often spans several weeks, encompassing diligent effort and occasionally, moments of frustration, as the engineer acquaints themselves with the unique installation requirements and formulates the precise blueprint. Without the expertise of a Dell solution engineer, the customer might spend months developing this solution on their own — a timeframe that is frequently a deal-breaker in this fast-paced industry.</p><h4><strong>So how can we address these challenges?</strong></h4><p>You guessed it — we’re using GenAI 😊</p><p>Let’s briefly overview our solution before diving deeper. We have two main models in our solution:</p><ol><li>A Q&amp;A model that helps answer questions about Dell’s documentation and internal technical resources.</li><li>A Code Autocompletion model that assists in creating blueprints.</li></ol><p>There’s also a Validation model in development, but we’ll skip the details for now. Additionally, our solution smoothly integrates into a development environment for testing purposes.</p><p>Now, let’s revisit our Dell video-analytics customer scenario to see how our tool can simplify and enhance their process. Instead of spending weeks with a Dell engineer, our customer’s engineer, whom we’ll call Nelly, uses our Q&amp;A model.</p><p>She asks it, “What are the steps to set up an OS on 5,000 cameras?” and gets the necessary steps in response. Next, Nelly starts writing the actual blueprint with the help of our second model. Once her blueprint is complete and validated, she can upload it to a development environment to test it on real devices. More on this in a bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*msq91l9Lsv24pbBvPshq6w.png" /></figure><h4><strong>Data and Modeling Insights</strong></h4><p>What kind of data and models are we leveraging for this project? First, let’s discuss what we didn’t use.</p><p>You might be wondering, “Why not just use Copilot or a commercial GPT for queries right off the bat?” We gave that a shot, but it fell short for our specific need to generate configuration files. This was mainly because our proprietary internal data wasn’t familiar to the model, leading us to the necessity of fine-tuning.</p><h4><strong>Our modeling Strategy:</strong></h4><p>For the Q&amp;A model, we’ve implemented a straightforward RAG architecture with a private GPT-3.5 instance. The data we’re utilizing comes from both internal and external sources, including Confluence and data on dell.com. Interested in RAG? There’s some excellent material out there for a deeper dive.</p><h4>Creating Blueprint/Configuration Files</h4><p>Our focus is on two primary objectives:</p><ol><li>Developing a code completion model that seamlessly integrates with your IDE.</li><li>Crafting a model capable of generating blueprints from descriptions.</li></ol><p><strong>Code Autocompletion Model:</strong></p><p>For the fine-tuning of our code autocompletion feature, we chose Startcoder. This model boasts 15 billion parameters and has been trained using a rich dataset from GitHub, encompassing over 80 programming languages, commits, issues, and Jupyter notebooks. As of May 2023, Startcoder has surpassed other large language models (LLMs) in popular programming benchmarks, featuring a context length of 8K — the largest at that time.</p><p>Startcoder’s capabilities are diverse, ranging from acting as a technical assistant and offering code autocompletion to modifying code based on instructions and explaining code snippets in plain language.</p><p><strong>Data Preparation for code autocompletion</strong></p><p>In preparation for fine-tuning with LoRA, we needed several thousand YAML configuration files, or blueprints, for training. Facing a shortage of these files, we employed augmentation techniques. This involved rearranging the order of various fields within the files, akin to shuffling the order of keys in a dictionary. Additionally, we tweaked the files by adding or removing certain optional fields to further diversify our dataset.</p><p>Our fine-tuning process with LoRA utilized 30,000 samples. It required some adjustments in parameters, like context size and LoRA rank, among others, to achieve our target outcome. We conducted manual evaluations of the model, with our engineers testing it across various use cases.</p><p><strong>Serving the model and integration with IDE:</strong></p><p>For the deployment of our model, we adopted the Text Generation Interface (TGI) provided by Hugging Face. This interface was particularly effective due to its compatibility with the llm-vscode extension from Hugging Face, facilitating a smooth integration with the development environment.</p><p>During our assessment phase, engineers attempted to create blueprints using the model’s assistance. This process was crucial in identifying the scenarios where the model excelled and pinpointing the types of blueprints that posed challenges for it. Through iterative refinements of both the data and the model’s parameters, we progressively enhanced its performance. This continued until we reached a point where our test engineers expressed satisfaction with the model’s capabilities.</p><h4>Generating Blueprints from Descriptive Language</h4><h4>Model Overview</h4><p>To refine our model for interpreting instructions, we utilized Code Llama. Code Llama belongs to a series of advanced language models for coding, built upon the Llama 2 architecture, known for its leading-edge performance among accessible models. Specifically, we employed the 13-billion-parameter version of Code Llama for instruction-based fine-tuning.</p><h4>Few-Shots Inference Approach</h4><p>In the process of generating blueprints from user descriptions, we initially experimented with a few-shots prompting technique. This approach involves providing the model with a set of examples in its input, followed by a request for the model to generate a new blueprint based on these examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*J4_CjT3ZYw3uoXkGwg9PTg.png" /></figure><p>This approach proved effective only in specific scenarios and didn’t offer the broad applicability we needed. Therefore, opting for fine-tuning became the logical next step.</p><h4><strong>Instruction fine-tuning</strong></h4><p><strong>Data Preparation</strong></p><p>To effectively fine-tune the model with instructions, it’s essential to prepare sample pairs in the following format:</p><ul><li>Instruction: A clear, natural language description specifying the desired configuration file.</li><li>Answer: The corresponding correct configuration file, or Blueprint.</li></ul><p>Here’s an example to illustrate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*ZRC7P245ry0BaW1MUR4gMg.png" /></figure><h4><strong>Creating the descriptions with gpt3.5</strong></h4><p>To compile our dataset, it was necessary to generate descriptions for each blueprint. For this, we utilized GPT-3.5 to create descriptions corresponding to each file. We experimented with various prompts to produce effective descriptions. Once we had our descriptions, we proceeded with the data preparation for modeling as previously outlined.</p><p>This development involved numerous iterations and thorough evaluations, as mentioned earlier. The goal was to refine the process until users of the tool genuinely felt its benefits, and we continually strive for further improvements.</p><h4>Winning the Global Dell Technologies Hackathon</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*G0keNuBNHrXjL_KRARSckw.jpeg" /></figure><p>The Dell ISG (Infrastructure Solutions Group) Global Hackathon is a yearly event, bringing together teams worldwide to address critical business, societal, or environmental challenges through innovative technology solutions. I had the privilege of leading a talented Israeli team, comprising brilliant engineers and data scientists. We joined forces in this hackathon, where over <strong>2,200 team members</strong> globally collaborated to present <strong>624 remarkable projects.</strong></p><p>Despite facing a difficult situation in Israel, our team showed incredible teamwork and commitment. And it all came to fruition — we won the Hackathon! This victory isn’t just a personal achievement; it’s proof of the amazing things that can happen when people work together, innovate, and never stop striving for the best. Now, we’re focused on growing our solution and turning it into a product for Dell’s customers worldwide.</p><p>Thank you for following our journey thus far. If you’re curious to learn more about our project or have any questions, feel free to reach out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55b8ed25b3a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tell LoRA I Love Her]]></title>
            <link>https://medium.com/@rshalom1984/tell-lora-i-love-her-0e49163577b1?source=rss-471c0ed81e43------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e49163577b1</guid>
            <category><![CDATA[finetune-llm]]></category>
            <category><![CDATA[machine-leanring]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[peft]]></category>
            <dc:creator><![CDATA[Rachel Shalom]]></dc:creator>
            <pubDate>Thu, 26 Oct 2023 10:20:10 GMT</pubDate>
            <atom:updated>2023-10-26T10:22:22.963Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, I aim to unravel the intricacies of Low Rank Adaptation (LoRA) and offer you a clear insight into why this method is effective. Rest assured, I’ll equip you with all the mathematical knowledge necessary to grasp LoRA comprehensively, but first, let’s delve into the necessity of LoRA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXyNwfCues_IldXB0LTymA.png" /></figure><h3><strong>The Challenges with Fine-tuning</strong></h3><p>Training LLMs is computationally intensive. Full fine-tuning requires memory, not just to store the model but various other parameters that are required during the training process.</p><p>Even if your computer can hold the model’s weights in memory, these can grow to hundreds of gigabytes for very large models (like Falcon 180B and llama 2 70B). In addition, you must also be able to allocate memory of optimizer states, gradients, forward activations and temporary memory throughout the training process.</p><p>These additional components can easily surpass the model’s size by several folds, swiftly surpassing the handling capacity of standard consumer hardware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/0*mysP4iPmTXtGPdaP" /></figure><p>In contrast to full fine-tuning — where every model weight is updated during supervised learning, parameter efficient fine-tuning methods like LoRA only updates small subsets of parameters.</p><p>As a result, the number of trained parameters/weights is much smaller than the original LLM’s weights. This makes memory requirements for training much more manageable.</p><p>In the next part of this blog, we’ll explore matrix properties like rank and factorization. If math isn’t your cup of tea (oh dear!), feel free to skip this section and focus on the more intuitive aspects of matrix rank.</p><h3>Why do we care about Matrix Rank? intuition</h3><p>Generally speaking, datasets can be represented as matrices. The rank of a matrix provides insights into the underlying relationships and dependencies within this data.</p><p>A higher rank indicates that the data has a more complex structure, with many linearly independent features or patterns. This means that the data contains a significant amount of unique information, and reducing its dimensionality without losing important insights can be challenging.</p><p>On the other hand, a lower rank suggests that the data is more compressible, meaning that there are dependencies and redundancies within the data. In practical terms, this can be leveraged for dimensionality reduction and data compression, allowing you to represent the data more efficiently without losing crucial information.</p><p>The same stands for LLMs. They are basically a set of very large matrices multiplied by each other. The parameters (matrices) represent the learned knowledge that these models have about the world and sometimes can be redundant-meaning it’s possible to represent the same knowledge with fewer parameters. <strong>And this is What LoRA is all about</strong>.</p><h3><strong>Refresh some Linear Algebra concepts</strong></h3><h4>Before we delve into the intricacies of LoRA, let’s take a step back to revisit some fascinating concepts related to vectors and matrices (exciting, isn’t it?)</h4><p><strong>1. Linearly Dependent Vectors</strong></p><p>A set of vectors in a vector space where at least one vector in the set can be expressed as a linear combination of others, or you can get one from a scalar multiplication of the other. For example, consider the following two vectors</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/132/0*lZRv6Kc5w6hfoArf.png" /></figure><p>These vectors are linearly dependent because you can express v2 as a scalar multiple of v1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/0*0N13i0mmiaNFC3zp.png" /></figure><p><strong>On the other hand, linearly independent vectors are vectors that “stand on their own“ </strong>and are not redundant in terms of their linear relationship. They contribute unique information to the vector space and none can be represented as a combination of others.</p><p><strong>2. Matrix Rank</strong></p><p>Matrix Rank is the number of linearly<strong> independent</strong> columns (or rows) in a given matrix. This is also called the dimension of the vector space generated by the matrix columns.</p><p>It can be proven that the number of linearly independent rows in a matrix is equal to the number of linearly independent columns.</p><p>So if we have a matrix V with m rows and n columns, then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/0*Z_19f1fg17s6QNdl.png" /></figure><p>Let’s look at the following example of a matrix V with 3 rows and 3 columns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/146/0*uxNFlRuCZPRRatmt.png" /></figure><p>You can notice that we can get the third row vector v3 by a combination of v1 and v2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/145/0*gXr_STCqzhnak0v9.png" /></figure><p>So vector v3 is linearly dependent on v1 and v2.</p><p><strong>Matrices can be classified based on their Rank:</strong></p><p><strong>A</strong>. <strong>Fully Ranked Matrix </strong>is when all rows or all columns of a matrix are linearly independent. In mathematical notation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/0*a5TJih0vp1qvuuR5.png" /></figure><p>B. <strong>Rank Deficient Matrix</strong> is when 1 or more rows in the matrix are a combination of other rows. In mathematical notation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*fk9YkZpSW9zmdjpW.png" /></figure><p>Here is a good example of rank deficient matrix, V, based on our example before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*lZRv6Kc5w6hfoArf.png" /></figure><p>You can clearly see that v2=2*v1 and hence</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/139/0*4BvR9GclXpGt8Ax-.png" /></figure><p><strong>C. The critical concept to grasp is that a Low Rank Matrix</strong> is characterized by having a <strong>significantly </strong>smaller number of linearly independent rows or columns compared to the total number of rows or columns. In mathematical notation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/0*PbgCmhPCb3vWWGX7.png" /></figure><p>Now that we’ve established an understanding of what a low-rank matrix is, let’s explore two more key concepts:</p><p><strong>3. Rank of matrix multiplication</strong></p><p>If we have 2 matrices A and B with rankA and rankB , respectively, then the rank of the matrix multiplication is constrained by the minimum of their individual ranks. In mathematical notation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/0*yiLxAb6UXrgXMMNM.png" /></figure><p>4. <strong>Low Rank Matrix Decomposition</strong></p><p><strong>A Matrix V with rank r can be represented as a multiplication of 2 smaller matrices:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/0*ppsLW1apXmh9-wnA.png" /></figure><ul><li><em>V </em>is the original matrix of size <em>m</em>×<em>n</em>.</li><li><em>C</em> is a matrix of size <em>m</em>×<em>r</em>.</li><li><em>F</em> is a matrix of size <em>r</em>×<em>n</em>.</li></ul><p>Then <em>V</em> can be factorized into the product of <em>C</em> and <em>F</em>, where <em>r</em> is the rank of V [here is a <a href="https://en.wikipedia.org/wiki/Rank_factorization#Existence">proof </a>that every finite matrix has rank decomposition, techniques like SVD (singular value decomposition) can be used for that].</p><p><strong>With that, we covered the most important concepts to understand LoRA So let’s dive into Lora and explore how it leverages these principles in the context of fine-tuning large AI models.</strong></p><h3><strong>Revisiting Fine-Tuning</strong></h3><p>Fine-tuning is the method through which we input data into our network, such as an LLM, and then adjust the weights using the weight updates obtained through backpropagation. Think of it as a training process, but we start with pre-trained weights as our initial point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LzmUl1iJOjw42-LdRfAaYw.png" /></figure><p>We begin by feeding data through the network. Afterwards, we compute weight updates (referred to as deltas) using the backpropagation process. These update weights are then merged with the existing network weights to yield new weights. We repeat this process iteratively until we achieve the desired outcome.</p><h3><strong>LoRA</strong></h3><p>So where does LoRA fit into this?</p><p>Numerous prior studies have demonstrated that pretrained models exhibit remarkably low intrinsic dimensionality. In simpler terms, they can be accurately represented using significantly fewer dimensions or parameters than they originally possess.</p><p>In the <a href="https://arxiv.org/pdf/2106.09685.pdf">LoRA paper</a>, they hypothesize that the change in weights during model adaptation (training) also have a low intrinsic rank/dimension. That means that these delta weights are<strong> low rank matrices</strong>, <strong>meaning that we don’t need all of the weights parameters to describe everything that’s going on</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4VTrRzjwj-3syUcAXzaRnw.png" /></figure><p><strong>So they use the matrix decomposition to represent this very large matrix as a potentially smaller combination of matrices.</strong></p><p>These matrices can be much smaller than the original matrix but represent the same thing.</p><p><strong>Why does this make sense?<br></strong>Large models are trained to capture the general representation of their domain (language for LLMs, audio + language for models like Whisper, and vision for image generation models). These models capture a variety of features which allow them to be used for diverse tasks with reasonable zero-shot accuracy. However, when adapting such a model to a specific task or dataset, only a few features need to be emphasized or re-learnt. This means that the update matrix (ΔW) can be a low-rank matrix.</p><h3><strong>Methodology</strong></h3><p>The technique is using decomposition to represent the change in weights <strong><em>ΔW </em></strong>as a product of 2 low rank matrices B and A that are lower in dimension.</p><p>Now, during fine-tuning all of the original model parameters are frozen and then a pair of small rank decomposition matrices are trained.</p><p>At inference the two low rank matrices are multiplied together to create a matrix with the same dimension as the original weights. You then add this to the original weights and replace them in the model with these updates values, and now you have a model with the updated values!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*94m--_wXV47QbZVolqOw0w.png" /></figure><h3>What are the main advantages of LoRA<strong>?</strong></h3><h4><strong>Significant Reduction in Computational Complexity.</strong></h4><p>Let’s assume we have a matrix of the base transformer model (as presented in the <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a> paper)</p><ul><li>The transformer’s weights have dimensions d*k=512*64</li><li>So each weights matrix have 512*64= 32,768 trainable parameters</li><li>If we use LoRA with rank = 8 we instead train 2 smaller dimension matrices A and B.</li><li>A has a dimension of r*k=8*64=512 parameters</li><li>B has a dimension of d*r=512*8=4096 trainable parameters</li><li>And there you go — 86% reduction in parameters to train!</li><li>This means you can (often) fine-tune your model on a single(!) GPU and avoid the need for a distributed cluster of GPUs.</li></ul><h4>Quick and Efficient Task-Switching</h4><p>LoRA allows for a pre-trained model to be shared and used to build many small modules for different tasks. This reduces the storage requirement and task-switching overhead significantly.</p><p>Consider a scenario in which you’ve trained a single foundational model for six different downstream tasks using LoRA. To transition between these tasks during both training and inference, all you need to do is substitute compact matrices.</p><p>Isn’t it intriguing that your customers can select their desired task during inference, and all you require is one foundational model that can exchange matrices based on your customers’ preferences? This is a remarkable advantage of using LoRA!</p><h3>Summary</h3><p>LoRA represents a major step forward in the efficient adaptation of LLMS. Through matrix decomposition, it effectively reduces the number of trainable parameters, enabling more efficient deployment and task-switching without compromising on model quality.</p><p>As a result, it becomes a valuable asset for organizations seeking to harness the capabilities of large language models in a cost-effective and efficient manner.</p><p>I hope you’ve found this blog post informative and that it has deepened your understanding of LoRA. Please don’t hesitate to share your thoughts and leave your comments. We highly value your feedback!</p><p>Resources:</p><p><a href="https://www.coursera.org/learn/generative-ai-with-llms">https://www.coursera.org/learn/generative-ai-with-llms</a></p><p><a href="https://huggingface.co/docs/peft/conceptual_guides/lora">https://huggingface.co/docs/peft/conceptual_guides/lora</a></p><p><a href="https://www.youtube.com/watch?v=lixMONUAjfs&amp;ab_channel=WesRoth">https://www.youtube.com/watch?v=lixMONUAjfs&amp;ab_channel=WesRoth</a></p><p><a href="https://www.ml6.eu/blogpost/low-rank-adaptation-a-technical-deep-dive">https://www.ml6.eu/blogpost/low-rank-adaptation-a-technical-deep-dive</a></p><p><a href="https://news.ycombinator.com/item?id=35288966">https://news.ycombinator.com/item?id=35288966</a></p><p><a href="https://medium.com/linear-algebra/part-8-linear-independence-rank-of-matrix-and-span-9f8af9bfd475#:~:text=To%20find%20if%20rows%20of,combination%20of%20other%20row%20vectors.&amp;text=Turns%20out%20vector%20a3%20is,A%20is%20not%20linearly%20independent">https://medium.com/linear-algebra/part-8-linear-independence-rank-of-matrix-and-span-9f8af9bfd475#:~:text=To%20find%20if%20rows%20of,combination%20of%20other%20row%20vectors.&amp;text=Turns%20out%20vector%20a3%20is,A%20is%20not%20linearly%20independent</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e49163577b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>