<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dreams Are Real on Medium]]></title>
        <description><![CDATA[Stories by Dreams Are Real on Medium]]></description>
        <link>https://medium.com/@dreamsarereal?source=rss-fedd731cbc22------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*HaVdtmMB7AC7L9QHjmhWdg.jpeg</url>
            <title>Stories by Dreams Are Real on Medium</title>
            <link>https://medium.com/@dreamsarereal?source=rss-fedd731cbc22------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:10:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dreamsarereal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding LoRA Training, Part 3: Block Weights]]></title>
            <link>https://medium.com/@dreamsarereal/understanding-lora-training-part-3-block-weights-967711816280?source=rss-fedd731cbc22------2</link>
            <guid isPermaLink="false">https://medium.com/p/967711816280</guid>
            <category><![CDATA[ai-art]]></category>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[stable-diffusion]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Dreams Are Real]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 01:11:41 GMT</pubDate>
            <atom:updated>2023-07-25T01:20:49.592Z</atom:updated>
            <content:encoded><![CDATA[<p>In order to understand block weights, we need to understand Stable Diffusion U-Net architecture first.</p><h3>U-Net</h3><p>If you’ve ever tried to research the topic, you definitely stumbled upon this U-shaped diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDbvfktbQl5HdQ5UrIpWQw.png" /><figcaption>U-Net as seen from a 100 miles away</figcaption></figure><p>An input image starts it’s way at the top left and after a transforming journey over a U-path is reborn at the top right.</p><p>I read a ton of articles on the subject in the last few days and the most accessible one for those of us who get easily discouraged by complex math is this one:</p><p><a href="https://towardsdatascience.com/you-cant-spell-diffusion-without-u-60635f569579">You can’t spell Diffusion without U</a></p><p>If you are not entirely sure what U-Net is, I encourage you to read it in full. It’s relatively short and totally worth it from start to finish.</p><p>I especially liked how <a href="https://medium.com/u/a99fbdf99073">Sairam Sundaresan</a> used the <a href="https://www.wikihow.com/Play-the-Telephone-Game">telephone game</a> as an example to describe the way U-Net blocks communicate with each other<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*v_0hELER1CwbScDszrE5yw.png" /><figcaption><em>Note the green arrows!</em></figcaption></figure><p>Assuming you’ve got a grip on it now, let’s see how <em>kohya-ss scripts</em> block weights correspond to U-Net blocks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*lb7Ww2wY0450eMNCHlwVTg.png" /></figure><p>If you ever looked at kohya-ss terminal window during training with block weights, you could’ve seen this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*EeGaqzxsmuDh7PCEXXdGRQ.png" /></figure><p>I hope the diagram above helps with understanding what <em>deeper</em> and <em>shallower</em> actually means! By changing block weights, we can affect deeper and shallower layers of U-Net separately.</p><h3>LoRA Blocks</h3><p>The official names of U-Net blocks correspond to kohya-ss blocks like this:</p><p><em>Down LR weights: </em><strong>IN00 to IN11</strong><br><em>Mid LR weights: </em><strong>M00</strong><br><em>Up LR weights: </em><strong>OUT00 to OUT11</strong></p><p>There is another key detail. When training Standard LoRA, we don’t affect the convolution layers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*0Sen6RN30j_KAyUr6pyHjg.png" /></figure><p>See <a href="https://github.com/kohya-ss/sd-scripts/issues/674#issuecomment-1646751324"><strong><em>this kohya-ss post</em></strong></a> for reference:</p><blockquote>The Stable Diffusion v1 U-Net has transformer blocks for IN01, IN02, IN04, IN05, IN07, IN08, MID, OUT03 to OUT11.</blockquote><blockquote>IN00, IN03, IN06, IN09, IN10, IN11, OUT00, OUT01, OUT02 don’t have transformer (which has Linear layers) and only have Conv2d layers.</blockquote><p>There is no reason to set custom weights for the “extra” blocks. They are simply inaccessible in Standard LoRA training.</p><h3>Block Roles</h3><p>There has been a lot of speculation on the roles of each block. You could’ve seen wild theories like these:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4tLBOahO7TjyWM8s8gDhbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hs9K4axZkzOTOC4uo3jByg.png" /></figure><p>If you’ve read the article mentioned above, you probably realise that it just doesn’t work like this.</p><p>Each block’s output depends on the input. Results differ a lot with closeup and long shots, for instance, as explained in this beautiful <em>LoRA Block Weights Guide </em>by <em>bdsqlsz </em>at Civitai:</p><p><a href="https://civitai.com/articles/76/bdsqlsz-lora-training-advanced-tutorial1lora-block-training">Civitai | bdsqlsz LoRA training Advanced Tutorial(1)：LoRA Block Training</a></p><h3>LoRA Block Weights A1111 extension</h3><p>Turns out you can set LoRA block weights at the generation stage by using <em>hako-mikan’s </em><strong><em>Lora Block Weight</em></strong> extension:</p><p><a href="https://github.com/hako-mikan/sd-webui-lora-block-weight">GitHub - hako-mikan/sd-webui-lora-block-weight</a></p><p>The <em>Overview</em> at the link above should be quite easy to understand at this point. Don’t miss the <em>XYZ plotting feature</em>!</p><p><strong>And the question is</strong>, does it make any sense to train with custom block weights if we can isolate them during inference?</p><h3>The setup</h3><h4>The Dataset</h4><p>Let’s try to teach a model to recognize the great photographer Roy Decarava’s style. He was inimitably imaging the lives of African Americans and jazz musicians in the middle of last century.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*NLUAE-kXw537H2W9efKqzg.png" /><figcaption>27 images total</figcaption></figure><p>My dataset has various aspect ratios this time, although all images are 1024px max on the longer side. I made a second dataset with the same images cropped to 512x512 to test the difference too, but I’ll leave the results for next time.</p><h4>Core Settings</h4><p><em>Source model:</em> <a href="https://civitai.com/models/13020">Avalon TRUvision</a><br><em>Resolution:</em> 512x512<br><em>Class images: </em>0<br><em>Img dataset repeats:</em> 1<br><em>Epochs:</em> 320<br><em>LR Scheduler:</em> Constant<br><strong><em>Text Encoder LR:</em> 0<em> (see explanation below)</em></strong><br><em>U-Net LR:</em> 1e-4<br><em>Dim/Alpha:</em> 128/128<br><em>Prior loss weight:</em> 1</p><h4>By The Way!</h4><p>If you often download or merge models, make sure to check out this <strong><em>Multipurpose toolkit for managing, editing and creating models</em></strong><em>:</em></p><p><a href="https://github.com/arenasys/stable-diffusion-webui-model-toolkit">GitHub - arenasys/stable-diffusion-webui-model-toolkit: A Multipurpose toolkit for managing, editing and creating models.</a></p><blockquote>Many models being distributed are quite bloated, most of their size being redundant or useless data.</blockquote><p>Here is the report for <em>Avalon TRUvision</em> I am using today, which is “<em>a custom blend of various models”:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*xwX-JrQmz4CnsyBmzN_ffw.png" /><figcaption>From <strong>3.89 GB</strong> to <strong>1.99 GB</strong>!</figcaption></figure><p>Using a lighter fully-functional version now!</p><h3>The Results</h3><p>Let’s compare the samples generated during training with custom and default block weights. The sample prompt was:</p><blockquote>black and white ird photo, ird film grain, ird lighting, woman, blond hair, headshot, sunflower field</blockquote><h4><strong><em>Default Block Weights</em></strong></h4><p>While we are here, let’s look at the effect of captions.</p><ul><li><em>My captions looked like this:</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*XS9rxuh_BWkBcNCwInzFlg.png" /></figure><p><strong>With captions<em>:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*_9g4Hr-FlXTMxmwA8Yz7Ig.png" /></figure><p>This LoRA is clearly overtrained, of course, and would benefit from lowering the learning rate and changing the scheduler, but that’s beyond the scope of today’s tests.</p><p><strong>No captions<em>:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*kpHGm-NX4qyPlnRPbFJsuQ.png" /></figure><p>Despite the fact that I wasn’t training Text Encoder (more on this below), the U-Net clearly learns how Roy Decarava’s women look like.</p><p>Note the absence of sunflowers early on with no captions training. This happened because U-Net only received “<em>ird photo”</em> as caption for each image in the dataset. It probably <em>assumed </em>that the source backgrounds (as well as everything else not specifically captioned) are a feature of an “ird photo” that deserves to dominate the rest of the prompt.</p><h4>Custom Block Weights</h4><p><strong><em>Down LR Weights at 0,1,1,0,1,1,0,0,0,0,0,0</em><br><em>All other block weights at 0</em></strong></p><ul><li><em>Training takes twice less time in this case</em></li></ul><p><strong>With captions:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*fuZUKKvdhMk5IIE5dcvZDg.png" /></figure><ul><li><em>Training with no captions doesn’t affect the result much.</em></li></ul><p>In case you’ve been wondering why I didn’t train the Text Encoder module, here is why.</p><p><strong>With captions + Text Encoder at 5e-5:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*NLkjFtYYLc_1PzDGe0rX2g.png" /></figure><p>All other settings are the same as above, but it couldn’t look more different! Adding text encoder to regular training with default block weights helps with pronouncing dataset features, but the difference is never this radical.</p><p>Here is another example:</p><p><strong><em>Down LR Weights at 0,1,1,0,0,0,0,0,0,0,0,0<br>Mid LR Weights at 0<br>Up LR Weights at 0,0,0,0,0,0,0,0,0,0,1,1</em></strong></p><p><strong>With captions:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*ZjFDNgMrVY16hpVWFtpnIA.png" /></figure><p><strong>With captions + Text Encoder at 5e-5:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*DvtzqI7F-d2656X2vH9l7A.png" /></figure><ul><li><em>The results are just as different when training without captions</em></li></ul><p>There is no mystery here. I am using block weights to train <em>select few </em>U-Net blocks while training <em>100% </em>of the Text Encoder. There is no way to isolate Text Encoder “blocks” in the same way because its architecture is completely different.</p><h4>XYZ Plots</h4><p>Let’s check the results on a few seeds with the same prompt and some help from <em>LoRA Block Weight </em>extension by applying <em>Down LR Weights at 0,</em><strong><em>1</em></strong><em>,</em><strong><em>1</em></strong><em>,0,</em><strong><em>1</em></strong><em>,</em><strong><em>1</em></strong><em>,0,0,0,0,0,0</em> to the images generated with LoRAs above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cWWBh3VUFLy5RyPYOCjxbA.png" /></figure><p>And here are LoRAs trained with Text Encoder, in case you are interested:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Zfy3dgl1i-7VUDmYEu0dQw.png" /></figure><p>Note that LoRA trained with <em>default block weights</em> is completely baked and generates images like these for the same seeds <em>without hako-mikan’s extension:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-WDFqks5nXZ9422toI6aqw.png" /></figure><p>While the other 2 LoRAs are borderline usable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJGwwevjz3BUwFr5P0MewQ.png" /></figure><p>Okay!</p><p>Wow. First of all, this is quite incredible.</p><p>The LoRA trained with the same weights I used for the generated images shines the most, of course, but I didn’t expect the baked default weights LoRA to produce any clear results.</p><p>In subsequent tests with hako-mikan’s extension and different seeds the LoRA trained with default weights proved to be more creative with compositions/poses while the custom block weights LoRAs were pretty fixed (still being a better match to the dataset):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hgc5l7xNsu_nexDNy8QiyA.png" /></figure><h3>A Recommended Workflow</h3><p>For the many reasons explained above there are no universal roles assigned to each U-Net block that we could apply to image generation. While their roles are surely set, they are very far from <em>composition</em>, <em>textures </em>or <em>color tones </em>we wish for.</p><p>Thanks to hako-mikan’s extension, there is a way to pinpoint which block weights are worth looking into by using an XYZ plot. Here is a matrix of the LoRA trained with default block weights:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bb5WCM8wwMDqgFxL6vl4DA.png" /></figure><blockquote><em>NONE:</em><strong> </strong>0,0,0,0,0,0,0,0,0,0,0,0,0,0,0<br><em>ALL:</em> 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1<br><em>INS:</em> 1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0<br><em>IND:</em> 1,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0<br><em>INALL:</em> 1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0<br><em>MIDD:</em> 1,0,0,0,1,1,1,1,1,1,1,1,1,0,0,0,0,0<br><em>OUTD:</em> 1,0,0,0,0,0,0,0,0,1,1,1,1,1,0,0,0,0<br><em>OUTS:</em> 1,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1<br><em>OUTALL:</em> 1,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1</blockquote><ul><li><em>First number (BASE) doesn’t apply to kohya-ss scripts weights. ‘Extra’ blocks are excluded.</em></li></ul><p>By looking at this matrix you can easily see the role of each of the block groups. You can make XYZ plots for individual blocks too, of course, but I guess it makes sense to get a general picture first. Here are my settings for the matrix above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*DYeCG6LXtzMCFzgG19s2Tw.png" /></figure><p>This is how the same matrix would look like for the LoRA trained with <em>Down LR Weights at 0,1,1,0,1,1,0,0,0,0,0,0</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e9lJ14nl4s8YuVCt_IUTaQ.png" /></figure><p>When you pinpoint your blocks of interest, you can set gradient neighbor weights for them like 0,1,0.75,0,0.5,0.25,0,0 etc. It’s another great trick for fine-tuning LoRA block weights.</p><h4>P.S.</h4><p>Did you know that you could add <em>--</em><strong><em>autolaunch</em></strong> to <em>COMMANDLINE_ARGS </em>line in A1111’s <em>webui-user.bat</em> to automatically open the browser window?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=967711816280" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding LoRA Training, Part 2: Offset Noise, Epochs and Repeats]]></title>
            <link>https://medium.com/@dreamsarereal/understanding-lora-training-part-2-offset-noise-epochs-and-repeats-c68b86c69da8?source=rss-fedd731cbc22------2</link>
            <guid isPermaLink="false">https://medium.com/p/c68b86c69da8</guid>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[text-to-image-generation]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[stable-diffusion]]></category>
            <category><![CDATA[ai-art]]></category>
            <dc:creator><![CDATA[Dreams Are Real]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 17:26:22 GMT</pubDate>
            <atom:updated>2023-07-20T17:26:22.752Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s talk about Offset Noise, which is supposed to be quite a crucial part of the training process.</p><p>Ever tried training a Stable Diffusion model against <em>a single solid black image</em>? Chances are you won’t get anything much better than this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*y_VY1PGM2WopFh3NaOLixw.png" /></figure><p>This image comes from <a href="https://www.crosslabs.org/blog/diffusion-with-offset-noise">the key article about Offset Noise</a>.</p><blockquote>It seems like not only does Stable Diffusion not have the ability to produce overly dark or light images out of the box, but it also <strong>can’t even learn to do it</strong>.</blockquote><p>It’s a fun read!</p><p>If you prefer a visual explanation, here is <a href="https://www.youtube.com/@lewingtonn">koiboi</a>’s version, amazing as always:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcVxQmbf3q7Q%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcVxQmbf3q7Q&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcVxQmbf3q7Q%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d55682f22caed6f504756f4f7029955/href">https://medium.com/media/6d55682f22caed6f504756f4f7029955/href</a></iframe><p>Remember how I said in <a href="https://medium.com/@dreamsarereal/understanding-lora-training-part-1-learning-rate-schedulers-network-dimension-and-alpha-c88a8658beb7"><strong>Part 1</strong></a> that a <em>weak handshake</em> between LoRA and the source model results in vivid colors? Well, I wasn’t entirely wrong there, but the reason for this color shift always happening early in training is not a ‘weak handshake’. It’s the issue that authors of Offset Noise algorithm are trying to solve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*FZx6A5EHTKdlg-swNfC7KQ.png" /></figure><p>Over the course of the training the brightness range of a LoRA is getting closer and closer to the average brightness of the dataset. That’s the idea of training and a good thing, of course, unless you want to generate low key or high key images.</p><p>Let’s try using black squares as <em>class images</em> and see what happens over the course of 18 epochs. I will be using <em>Cosine with 18 restarts</em> as LR scheduler.</p><p>Before we start, this is what I had with native class images (see <a href="https://medium.com/@dreamsarereal/understanding-lora-training-part-1-learning-rate-schedulers-network-dimension-and-alpha-c88a8658beb7">Part 1</a> for details and settings):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*8UW3G0y6kvuU0k7q7OgE7g.png" /></figure><p>Now let’s change class images to black squares.</p><p><strong>Sample prompt is the same:<br><em>dfw woman, headshot, sunflower field</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*NnckVFZS0beZQ3C4RDrgTw.png" /></figure><p>The brightness range has clearly extended. I got brighter highlights and darker shadows, but the quality of character training predictably went down. This was done with 10 repeats of 14 training images and 140 repeats of 1 black square class image to get the same number of epochs as in the first test.</p><p>If I had 14 images with no repeats per epoch (which is more logical) and 14 repeats of the black square to balance the training dataset, this what I’d get over 18 epochs <em>(only 252 steps)</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*xl98uDdUvkoav8ELdM0cRA.png" /></figure><ul><li><em>More on epochs and repeats in a separate section below!</em></li></ul><p>To retain the training quality and brightness range, I guess I can add <em>‘low key’</em> and <em>‘high key’</em> to the class images prompts. It’s such an obvious trick that there is no need to test the results.</p><p>For now, let’s try a different sample prompt:<br><em>dfw woman, headshot, sunflower field</em> <strong><em>at night</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*4NWCEedkRLF-aymZ7MrkJg.png" /></figure><p>This doesn’t really look like night, so yeah, it’s a clear example of the average brightness issue.</p><p>Let’s add <strong><em>Offset Noise 0.12:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*7___3anhTffUS0qfLSbucg.png" /></figure><p>The night is here! Although it looks like the training set was affected in more ways than the average brightness change.</p><p>Let’s try <a href="https://wandb.ai/johnowhitaker/multires_noise/reports/Multi-Resolution-Noise-for-Diffusion-Model-Training--VmlldzozNjYyOTU2"><strong><em>Multi-Resolution Noise</em></strong></a><strong><em> at 8 iterations and 0.45 discount </em></strong><em>(a bit of an overkill) </em>instead of the original Offset Noise:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*0DUYR5Q8FXE2q5b2Da5o9A.png" /></figure><p>Okay, I got it. I hope you did too!</p><p>There is probably a solid middle ground for noise settings, but I can’t say that I’m a fan of the approach.</p><p>It’s good to be aware of the average brightness issue, especially if you are training a low (or high) key style, but I’m not sure adding the noise is worth sacrificing the textures quality during the training.</p><p>I’d rather be vigilant of my dataset brightness for both training and class images. And I can always <a href="https://civitai.com/models/8765">fix the brightness on the generation stage</a> too.</p><h3>Epochs and repeats</h3><p>It seems that there is a lot of confusion around kohya-ss folder structure, repeats and epochs. <em>As kohya-ss explained </em><a href="https://github.com/kohya-ss/sd-scripts/discussions/182#discussioncomment-4946693"><em>here</em></a>, there are two reasons for repeats:</p><p><strong>1. Balancing the number of training images with class (reg) images.</strong> Usually there are more class images than training images, so it is required to repeat training images to use all regularization images in the epoch.</p><ul><li><em>For instance, if you have 10 training images and 100 class images, you repeat them 10 times (10x10 = 100), to compare them to all class images within each epoch.</em></li></ul><p><strong>2. Control ‘weight’ over folders.</strong> If you have a mix of high quality images and low quality images, you can set higher number of repeats for high quality images and lower for low quality.</p><ul><li><em>It doesn’t make sense to repeat training images more times than needed for balance or weight within one epoch, otherwise you lose the flexibility of using epochs in your training.</em></li></ul><p>Kohya-ss scripts default settings (like <em>40 repeats </em>for the training dataset or <em>Network Alpha at 1</em>) are not ideal for everyone. Most of these settings are at the very low values to avoid issue reports from users with weaker systems. Sure enough, it makes sense to fine-tune them to your hardware and training needs.</p><p>Make sure to check if you really need AdamW8bit set as optimizer, take a look at Precision settings, the number of CPU threads and everything else. You might save yourself days of training in the long run by speeding things up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c68b86c69da8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding LoRA Training, Part 1: Learning Rate Schedulers, Network Dimension and Alpha]]></title>
            <link>https://medium.com/@dreamsarereal/understanding-lora-training-part-1-learning-rate-schedulers-network-dimension-and-alpha-c88a8658beb7?source=rss-fedd731cbc22------2</link>
            <guid isPermaLink="false">https://medium.com/p/c88a8658beb7</guid>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[stable-diffusion]]></category>
            <category><![CDATA[ai-art]]></category>
            <dc:creator><![CDATA[Dreams Are Real]]></dc:creator>
            <pubDate>Tue, 18 Jul 2023 17:47:56 GMT</pubDate>
            <atom:updated>2023-07-18T17:47:56.797Z</atom:updated>
            <content:encoded><![CDATA[<p>As the title suggests, this is an intermediate level blog post!</p><p>If you’d like to catch up, there are a few great videos for beginners. Here is <a href="https://www.youtube.com/watch?v=70H03cv57-o">one</a>, and there is <a href="https://www.youtube.com/watch?v=7m522D01mh0">another</a>. Well-structured written guides are also available:</p><ul><li><a href="https://rentry.co/dummylora">Dummy local LoRA usage and local training setup guide (Windows, Nvidia)</a></li><li><a href="https://rentry.co/59xed3">THE OTHER LoRA TRAINING RENTRY</a></li></ul><p>I am using <strong>kohya-ss</strong> scripts with <a href="https://github.com/bmaltais/kohya_ss">bmaltais GUI</a> for my LoRA training, not <a href="https://github.com/d8ahazard/sd_dreambooth_extension/">d8ahazard dreambooth A1111 extension</a>, which is another popular option. From my experience, bmaltais implementation is maintained better and offers more parameters to work with.</p><p>Readme docs for kohya-ss scripts at GitHub are worth reading as well, of course:</p><p><a href="https://github.com/darkstorm2150/sd-scripts#links-to-usage-documentation">GitHub - darkstorm2150/sd-scripts: This is a ChatGPT-4 English adaptation of the original documents by kohya-ss</a></p><p>The goal today is to understand the role of <strong>Network Rank (Dimension) and Network Alpha </strong>parameters in character training. There will be quite a few takeways on learning rate schedulers and class (regularisation) images along the way too!</p><h3>The setup</h3><h4>Learning Rate Schedulers</h4><p>Learning rate is the key parameter in model training. The bigger the number, the faster the training and the more details are missed in the process. It is well explained visually <a href="https://youtu.be/2ityl_dNRNw?t=711">here</a>.</p><p>A <em>constant</em> training rate is one option. There are a few more available in the GUI at the time of writing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*nKsui5WPHCpTXAllbXAjBA.png" /></figure><p>An optimal training process involves a variable learning rate. You might want your model to recognize the key features of your dataset by learning fast at the beginning and then gradually decrease the learning rate for smaller details. A training process of this kind would look like this on a graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*2evAlDioitirs2gB_HHJCA.png" /></figure><p>Here is what happens in the training process when you choose <em>Constant</em>, <em>Linear</em>, <em>Cosine</em> or <em>Cosine with restarts </em>schedulers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*YePa9oCiYew9cUlC4zRE7Q.png" /></figure><p>Warmup steps for <strong>Constant</strong> and <strong>Linear</strong> are set in % and allow for gradual increase of the learning rate during the first X% steps of the training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*Dkzmu7E0j9NlVArW4GhNpA.png" /></figure><p>This is possibly beneficial for datasets with a ton of intricate details that might get lost in the sauce if you start too fast.</p><ul><li><em>Note that you can set LR warmup to 100% and get a gradual learning rate increase over the full course of the training.</em></li></ul><p><strong>Cosine</strong> needs no explanation.</p><p><strong>Cosine with restarts</strong> is actually <strong>Cosine with <em>warm </em>restarts</strong>, which means that only the ‘good weights’ are used as the starting point of each restart <em>(in contrast to a </em><strong><em>cold </em></strong><em>restart where a new set of random numbers is used)</em>.</p><p>The amount of restarts can be set in <strong>LR number of cycles </strong>at the <strong>Advanced Configuration</strong> tab. <em>By default it is equal to the </em><strong><em>number of epochs</em></strong><em>.</em></p><p><strong>Polynomial</strong> LR scheduler works like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*InEe9gno34BXgHwg84gQ6g.png" /></figure><p>Its power can be set in <strong>LR power</strong> at the<strong> Advanced Configuration </strong>tab.</p><p><strong>Polynomial</strong> curve is more intense compared to <strong>Cosine </strong>scheduler, so it could be beneficial when your dataset doesn’t have many distinct features to be learned, but you want the training to be more efficient compared to <strong>Constant </strong>scheduler.</p><p><strong>Adafactor </strong>is probably an adaptive LR Scheduler (??), which probably pairs with Adafactor optimizer. More on adaptive optimizers below.</p><h4>Optimizers</h4><p>A quick word about optimizers before this post gets too long.</p><p>I haven’t looked into adaptive optimizers. LoRA training process has way too many volatile variables already, which makes it difficult to pinpoint the areas worth debugging. Adding a black box like <em>adaptive optimizer</em> would probably make things even more cloudy. I might be wrong, but that’s where I am at the moment.</p><p>As for the non-adaptive ones,<strong> Lion</strong> optimizer (actually <a href="https://www.sciencedirect.com/science/article/pii/S2288430015000524">inspired by lions and their cooperation</a>!) looks fascinating, but I haven’t found a common ground with it so far.</p><p>I will be using <strong>AdamW</strong> for my tests today.</p><h4>The dataset</h4><p>Let’s teach <a href="https://civitai.com/models/79754/reliberate"><strong>Reliberate</strong></a> source model to recognize someone it has no idea about.</p><p>Turns out it has no idea about Elina Löwensohn’s character Sofia Ludens from Hal Hartley’s 1994 film <em>Amateur</em>, nor does it know anything about Belgian model and actress Delfine Bafort.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tp2a_cL-r9229q9tT4LzzQ.png" /><figcaption>‘Sofia Ludens’ as seen by vanilla Reliberate</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4KGJKjLljXND2UPfinjiaA.png" /><figcaption>‘Delfine Bafort’ as seen by vanilla Reliberate</figcaption></figure><p>I will use two small datasets (in separate tests) to balance the results of my tests.</p><p><strong>DB Dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5NkY2y4yYLFCvgd_o6yUQ.png" /></figure><p>This dataset is near perfect for face training if you are not looking to generate a range of emotions. The facial expressions are stable, which makes it easier to recognize the person, the lighting is solid and there are plenty of different details to caption out:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*NDboo6v1n76fbTH0Z0scug.png" /></figure><p><strong>dfw</strong> is a single token character combination I will be using to identify the subject in training.</p><ul><li><em>You can get a list of all single tokens up to 4 characters </em><a href="https://github.com/2kpr/dreambooth-tokens/blob/main/all_single_tokens_to_4_characters.txt"><em>here</em></a><em>. Ideally check if your pick has any specific meaning for the source model.</em></li></ul><p><strong>SL Dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bnn7T952NZou8E4HMfzkrw.png" /></figure><p>SL is much more difficult. The lighting on most of the images is flat (higher contrast is better for subject recognition). There are too many angles and facial expressions for such a small dataset. On top of that I only captioned out the backgrounds and the phone handset, so that the model learns the character as a whole, dress and necklace included.</p><h4>Class images</h4><p>I generated 10 class (regularisation) images for each training image in both datasets. All images were generated using the source model (Reliberate) with the same captions as the training images, only the instance token was replaced with ‘woman’.</p><p>Properly captioned and generated class images help with making the training process more efficient by spotlighting the right blocks for the model.</p><h4>Folder Structure</h4><p>I have 13 training images in <strong>SL</strong> <strong>dataset </strong>and I want them to be trained alongside all 130 class images over the course of each epoch, so I am going to repeat them 10 times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*VSDq48vtjgnCDVqiCbiXzw.png" /><figcaption>I am using ‘slo’ as instance token for SL dataset</figcaption></figure><ul><li>Note that the folder name is ignored by kohya scripts when caption files are present.</li></ul><h4>Core Settings</h4><p><em>Source model:</em> Reliberate<em><br>Resolution: </em>512x512<em><br>Epochs: </em>18<em><br>LR Scheduler: </em>Cosine<em><br>Text Encoder Learning Rate: </em>5e-5<em><br>Unet Learning Rate: </em>1e-4<em><br>Prior loss weight: </em>1</p><p>I will be training for 18 epochs, which gives me <strong>2,340</strong> steps for SL dataset (18x130) and <strong>2,520</strong> steps for DB (18x140).</p><h3>The Results</h3><h4>DL dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*eVQGPH5JN4IcVEzLg-0UJA.png" /><figcaption><em>Prompt: dfw woman, medium headshot, wheat field</em></figcaption></figure><p>Same seed was used for these generations. Not much is clear apart from the fact that less details are learned with <em>Dim 8</em> and colors are a bit more vivid with <em>Alpha 1</em>.</p><p>Let’s look under the hood of the training process. I was generating samples at the end of each epoch with the same prompt (<em>dfw woman, headshot, sunflower field) </em>and the same seed.</p><ul><li><em>You can generate samples with the same seed by adding </em><strong><em>--d 1234</em></strong><em> at the end of your sample prompt. Any other number instead of 1234 will do, of course.</em></li></ul><p><strong>Dim 128, Alpha 128:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*FT8ckSmXDu4hQlhlrKqj8Q.png" /></figure><p><strong>Dim 128, Alpha 1:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*5Av9IKwcnHAL2zjtRz5uPw.png" /></figure><p><strong>Dim 8, Alpha 8:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*8wDduQBuP8J4dg0IiH4jQQ.png" /></figure><p><strong>Dim 8, Alpha 1:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*YTIbY-MOjSu7cTjbHJr1wQ.png" /></figure><p>Okay! This is more interesting.</p><p>Looks like <em>Network Rank (Dimension)</em> is the number of <em>features</em> to be trained, while <em>Network Alpha</em> is how many of them are allowed to alter the source model.</p><p>The colors seem to get more vivid when LoRA and the source model are not properly aligned, as if their handshake is unstable. It’s happening at the first steps of every test and doesn’t really go away when Alpha=1.</p><p>To make sure <em>Dim 8</em> is actually <em>limited in trained features</em>, let’s try increasing both learning rates.</p><p><strong>Dim 8, Alpha 8, Text Encoder LR 1e-4, Unet LR 5e-4:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*Mn36Hq_xqoJ3EvHu5f1PWg.png" /></figure><p>Much better than with the original learning rates, and yet still limited in trained features.</p><p>The key takeway here is that we need to increase the learning rate when we limit <em>Network Dimension</em>. Our tool is dull and small with lower <em>Dim</em>, so we need to work faster to achieve something.</p><p>No wonder kohya scripts default settings are at <em>Dim 8, Alpha 1</em>. It promotes a speedy training process, but the dataset requirements must be quite strict in this case.</p><p>Let’s look at a couple more sample sets.</p><p><strong>Dim 128, Alpha 128, <em>Text Encoder LR 0</em>, Unet LR 1e4:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*G-Pjvpf0qqK-tLBisuh2Pg.png" /></figure><p><strong>Dim 128, Alpha 128, TE LR 5e5, <em>Unet LR 0</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*-fOaPoooEuzymAZ5VDjgGA.png" /></figure><p>As expected, lol.</p><p>Okay, it looks like we are stalling at the last ~ 6 epochs. What if we change the scheduler from <em>Cosine</em> to <em>Cosine with restarts</em>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*sin3aIIDyv9dlPp1gLMk3Q.png" /></figure><p><strong>Dim 128, Alpha 128, <em>Cosine with 3 restarts (every 6 epochs)</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*YjLTBGfRUKgG1AqfsZ6jdw.png" /></figure><p>Helpful, but not too much.</p><p><strong>Dim 128, Alpha 128, <em>Cosine with 18 restarts (every epoch)</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*LpOFmVz8GfuU6VgpN0L5cA.png" /></figure><p>Quite helpful, actually! Good to know.</p><p>Before I get to the <em>SL dataset</em>, here is an interesting mistake to look at. At the beginning of my tests I accidentally trained LoRA on a different source model (<a href="https://civitai.com/models/82984">Paragon</a> instead of Reliberate) using the class images generated with Reliberate.</p><p><strong>Dim 128, Alpha 128, <em>Paragon source model </em>with <em>Reliberate class images</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*02ZVnAU2pR_vr95VCE7KQg.png" /></figure><p>The colors are more vivid here, again, possibly an indication of a weaker handshake between LoRA and the model. The symmetrical sunflowers in the last epochs <em>(seen at the original Dim128/Alpha128 above as well)</em> got much more ridiculous here.</p><p>This sample set also suggests that training is more efficient with native class images <em>= generated by the source model</em>. A few more examples:</p><p><strong>Dim 128, Alpha 1, <em>Paragon source model </em>with <em>Reliberate class images</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*pssBDig6046aIF1-jsMjKg.png" /></figure><p><strong>Dim 8, Alpha 8, <em>Paragon source model </em>with <em>Reliberate class images</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*U-o7qlr0liQyw2DCkwrO8A.png" /></figure><p><strong>Dim 8, Alpha 1, <em>Paragon source model </em>with <em>Reliberate class images</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*55JMpQFnAyF4VJiq5n35xw.png" /></figure><p>Yeah.</p><h4>SL dataset</h4><p>This is where things get even more interesting!</p><p>In case you forgot how this dataset looked like, here is a reminder:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*FL3ih7nZMR9EwpdGzQ_xWA.png" /></figure><p>Let’s start with a <em>slo woman, headshot, sunflower field</em> over the course of 18 epochs with original core settings.</p><p><strong>Dim 128, Alpha 128:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*a_FA_pRO7JK6v2JwTKNa6A.png" /></figure><p>Hello? I knew it will be challenging, but I still expected a bit more than this by the end of the training process. The 1st epoch sample was so promising, and then we didin’t get anywhere somehow.</p><p>For the sake of experiment I’d rather keep the number of steps as is. Let’s try changing the Dimension instead.</p><p><strong>Dim 256, Alpha 256:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*rbTvnxDySXP2v9IXTAdNIQ.png" /></figure><p>You can see that it’s trying, probably needs a more intense learning curve?</p><p>Feels like the right time for <em>Polynomial!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*QqghRZOYfe7ua3hcsod-GA.png" /><figcaption>Approximate visualization of Polynomial curve with a power of 0.1</figcaption></figure><p><strong>Dim 256, Alpha 256, <em>Polynomial with LR Power of 0.1</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*K3c1Cdp3ViDUagzBXIF1bg.png" /></figure><p>Right! Got pretty overbaked at the end, but the direction is good.</p><p>They say that <em>dataset is the king of the training</em>, and these samples definitely reflect that. The mix of facial expressions in SL dataset of poorly captioned 13 images (only the backgrounds and a phone handset excluded) combined with a mix of high contrast and flat lighting result in a nearly impossible training task for the model.</p><p>To wrap things up, let’s see if <em>Cosine with restarts</em> could save us here.</p><p><strong>Dim 256, Alpha 256, <em>Cosine with 18 restarts</em>:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*h_Bl-AZFQ1FfyuP6qoxx4w.png" /></figure><p>No, not really. Better than everything else though!</p><p>I hope you’ve got a better understanding of balancing Learning Rates and Schedulers with Network Dimension and Alpha settings depending on your dataset and training goals. Feel free to reach out in the comments to discuss the key takeaways and testing practices.</p><p>My next adventure is training a style with block weights. Subscribe to stay on board!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c88a8658beb7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>