<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arxiv Insights on Medium]]></title>
        <description><![CDATA[Stories by Arxiv Insights on Medium]]></description>
        <link>https://medium.com/@xanderst?source=rss-8e7e4128ff60------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0Tq4-5ehkphYjlpc.jpg</url>
            <title>Stories by Arxiv Insights on Medium</title>
            <link>https://medium.com/@xanderst?source=rss-8e7e4128ff60------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:53:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xanderst/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Eden, the garden of artificial delights]]></title>
            <link>https://xanderst.medium.com/eden-the-garden-of-artificial-delights-5b01a2714b15?source=rss-8e7e4128ff60------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b01a2714b15</guid>
            <category><![CDATA[text2image]]></category>
            <category><![CDATA[ai-art]]></category>
            <category><![CDATA[eden]]></category>
            <category><![CDATA[stable-diffusion]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Arxiv Insights]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 00:42:27 GMT</pubDate>
            <atom:updated>2023-08-11T15:02:04.889Z</atom:updated>
            <content:encoded><![CDATA[<p>We are thrilled to finally launch <a href="https://eden.art/"><strong>Eden</strong></a>, a collaborative, social platform for artists, creative technologists and machine learners to deploy and interact with generative AI models. Eden is built around an <a href="https://github.com/abraham-ai">open-source</a> ecosystem of tools which aims to facilitate discovery and collaboration among creatives, as well as to lower the barrier to entry for artists looking to leverage AI. For all the latest news and updates, join our <a href="https://discord.gg/4dSYwDT">Discord</a> and follow us on <a href="https://twitter.com/Eden_Art_">Twitter</a>.</p><p>This blogpost will focus on the practical usage of our platform, a more in depth write-up of this project’s vision and aspirations wrt open provenance for generative AI can be found <a href="https://medium.com/@genekogan/provenance-on-eden-d80dacce572a">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*WEJM3Qy6vid3-E9aLZcGFA.gif" /><figcaption>Videoloop created with Eden Remix + Real2Real</figcaption></figure><p>This post will overview the current features of the Eden App, which can be interacted with directly from the browser, no installation or special hardware needed. For now, our app is only optimized for desktop, smartphone support is on the way!</p><h3>Create an account</h3><p><a href="https://app.eden.art/">Creating an account</a> is super easy and takes just one minute, you can either connect with an existing MetaMask wallet or just connect with your gmail or phone number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRHiHFrWre0W088u9aK95g.png" /><figcaption>We support a variety of ways to connect to Eden</figcaption></figure><h3>Welcome to the garden</h3><p>Once you’ve signed into your account you’re ready to start browsing / creating! You can take a look at our content feeds or go directly to the “create” tab to make your own!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gi3Ws6GJJesKN9G3YwYHcw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EIf0xTSw0D5iq8nSK6V3kw.png" /><figcaption>Overview of our different creation endpoints</figcaption></figure><h3>Overview</h3><p>Before diving into each endpoint separately, lets do a quick overview of what each one does:</p><ul><li><strong>Create</strong> is our “text-to-image” pipeline, allowing you to create images from prompts using SDXL (StableDiffusion XL)</li><li><strong>Interpolate</strong> is an extension of that where you enter multiple prompts and get an interpolation video back that morphs through those prompts</li><li><strong>Real2Real</strong> is like Interpolate, but instead of prompts, this endpoint works off images only. You upload a sequence of images and the model will generate a smooth video morph between your images!</li><li><strong>Remix</strong> let’s you upload a single image and create variations of it, no prompts needed.</li><li><strong>Interrogate</strong> let’s you upload an image and get back a matching prompt that describes that image.</li><li><strong>Lora train</strong> let’s you upload a few (1–5) images of a specific person or object and teaches a custom StableDiffusion model what that person looks like. This takes about 15 minutes and, after training, you can use that custom lora model in all the above endpoints to make artworks of your favorite person! A VERY powerful tool indeed!</li><li><strong>TTS</strong> is a customizable text-to-speech model. Upload a few audio samples of a voice, write some lines of text and have the AI model produce customized audio. <em>Always think twice before using these features, digital consent is an important emerging ethical topic we should all be mindful of.</em></li><li><strong>Wav2Lip</strong> takes an audio file and animates an image or video of a face to make it speak the audio.</li><li><strong>Complete</strong> calls OpenAI’s GPT models to autocomplete a given text prompt.</li></ul><p>Now, lets go into each of these for some extra juicy details. After all, getting good results with AI often requires a bit of understanding about what goes on under the hood.</p><h3>Create</h3><p>Create is our text-to-image endpoint, powered by <a href="https://stability.ai/blog/stable-diffusion-public-release">StableDiffusion</a>. Set your desired image resolution, pick one of the checkpoints (these are all variations of StableDiffusion trained on different datasets and will yield different results, even for the same prompt), enter your prompt and hit create, simple as that!</p><p>If you’re the first person to trigger a creation job in a while, it is possible that our backend will spin up a new gpu-box for you, which might take a few minutes. Once a gpu is up and running, image creations shouldn’t take more than 15 seconds.</p><h4>Optional settings</h4><p>Every one of our endpoints has a dropdown “optional settings” that you can adjust if you know what you’re doing.</p><ul><li>For example, “<strong>upscale</strong>” wil upscale the resolution of your generated image by the given factor. If you want very HD images, upscaling is generally better than simply rendering at higher starting resolutions (width and height) because the generators are usually trained for a specific resolution and going too far beyond that can create repeating artifacts in the image, but feel free to experiment here!</li><li>“<strong>init image</strong>” let’s you upload an image that the AI model will use as a color and shape template to start drawing from. This allows much more control over what the final image should look like.</li><li>The ‘<strong>init image strength</strong>’ controls how heavily this init image influences the final creation, a good first value to try is 0.35</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNxU4jE0mzQvO8QAgZup-A.jpeg" /><figcaption>Init image (left) and resulting creation (prompt: portrait of a beautiful, young humanoid robot wearing a veil, digital painting, artstation, concept art, sharp focus, cinematic lighting, cgsociety)</figcaption></figure><h3>Interpolate</h3><p>Interpolate lets you create smooth interpolation video’s by entering a sequence of prompts. This allows you to create simple, linear video narratives and is fully compatible with custom concept training (LORA) explained below. Here’s a simple videoloop between the following prompts:</p><ul><li>“a single lone sprout grows in a barren desert, the horizon is visible in the background, low angle 8k HD nature photo”</li><li>“a lone sappling growing in a field of mud, realistic water colour”</li><li>“a giant old Tree of life, beautiful, intricate, 8k highly professionally detailed, HDR”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*1tHEPNu3WkQax1fRX13qCA.gif" /><figcaption>Lerping between 3 consecutive input prompts</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBo3VZCjDhGI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBo3VZCjDhGI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBo3VZCjDhGI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/54433c8d29d3f75952a7108a03b0416d/href">https://medium.com/media/54433c8d29d3f75952a7108a03b0416d/href</a></iframe><h3>Real2Real</h3><p>Real2Real is an algorithm we’re pretty proud of. It essentially does the same as lerp, except that here, the input is not a sequence of prompts, but a <strong>sequence of arbitrary images </strong>(currently .jpg imgs only). The algorithm will then create a smoothed video interpolation morphing between those real input images, no prompt engineering required.</p><blockquote><strong>Real2Real accepts ANY input image, so you can eg import images from MidJourney, use photographs, sketches, video frames, …</strong></blockquote><p>Below is an example of a Real2Real morphing between the following input images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1aH3FaszbFGJF5oIXk0xVw.jpeg" /><figcaption>(real) input images</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5a-hcE8OfQo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5a-hcE8OfQo&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e7d847c01f0485252dd76ffbfd668f1/href">https://medium.com/media/0e7d847c01f0485252dd76ffbfd668f1/href</a></iframe><p>Note that while Real2Real accepts litterally any input image, the quality of the interpolation will depend on how well the generative model can represent the input images. Eg StableDiffusion was not particularly trained on faces and so Real2Real tends to give underwhelming results on face interpolations.</p><p>Like our other endpoints, Real2Real has a few customization parameters that can dramatically affect the results from this algorithm:</p><ul><li><strong>FILM iterations</strong>: when set to 1, this will post-process the video frames using <a href="https://github.com/google-research/frame-interpolation">FILM</a>, dramatically improving the smoothness of the video (and doubling the number of frames).</li><li><strong>Init image min strength</strong>: the minimum strength of the init_imgs during the interpolation. This parameter has a significant effect on the result: low values (eg 0.0–0.20) will result in interpolations that have a longer “visual path length”, ie: more things are changing and moving: the video contains more information at the cost of less smoothness / more jitter. Higher values (eg 0.20–0.40) will seem to change more slowly and carry less visual information, but will also be more stable and smoother. <br>→ Experiment and see what works best for you!</li><li><strong>Init image max strength</strong>: the maximum strength of the init_imgs during the interpolation. Setting this to 1.0 will <strong>exactly</strong> reproduce the init_imgs at the keyframe positions in the interpolation at the cost of a brief flicker (<em>due to not being encoded+decoded by VQGAN</em>). Setting this to lower values (eg 0.70–0.90) will give the model some freedom to ‘hallucinate’ around the init_img, often creating smoother transitions. Recommended values are 0.90–0.97, experiment!</li></ul><h3>Remix</h3><p>Remix does exactly what you think it does: it <strong>takes an input image and creates a variation of it</strong>. The most important parameter here is:</p><ul><li><strong>Init image strength</strong>: controls how much influence the init image has over the final result. Setting this to 0.0 will produce a remix that is entirely based on the ‘guessed prompt’ for the image and not influenced at all by the actual colors / shape of the input image.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTfcLAVyKqm2ExIEL_pQeQ.png" /><figcaption>initial logo draft for <a href="https://remix-alias.vercel.app/">https://remix-alias.vercel.app/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_OPXRLWMiBIMVuhd-1BfQ.jpeg" /><figcaption>Various remixes of the Alias logo using Eden Remix</figcaption></figure><h3>Interrogate</h3><p>This endpoint takes an input image and returns a prompt that describes the input image (this is what powers Remix). You can then use this prompt in other endpoints like create or lerp to generate similar content.</p><h3>LORA Train</h3><p><a href="https://github.com/cloneofsimo/lora">LORA</a> is a fascinating technique that lets you teach new visual concepts to the AI model by using just a few example images. Let’s start by showing WHAT it can do, before explaining HOW to do it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rouHoigiLIKq9SdFlOXJxQ.jpeg" /><figcaption>4 training images, cropped so that only a single face is present in the image.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOZ3XAVMsyWkKcaC7a6-eQ.jpeg" /><figcaption>Creations using a LORA trained on the above images</figcaption></figure><p>Unlike many apps out there that simply give you a bunch of cool looking images with your face on them, our tools allow full customization and promtability with the custom concept:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5PPsg7nmEEBvfAQoUBxeSA.jpeg" /><figcaption>LORA models are fully promptable: “&lt;xander&gt; sitting on the iron throne” (top left) || “&lt;xander&gt; as the emperor in front of the colloseum” (top right) || “&lt;xander&gt; as the pope” (bottom left) || “&lt;xander&gt; taking a shower” (bottom right)</figcaption></figure><p>Additionally, once a LORA model is trained you can use it in any of our other endpoints (create, lerp, real2real, remix, …)</p><h4>How to train a good LORA model</h4><p>Training a good LORA model is a bit of a subtle art and requires some experimentation / iteration to get it right. Unfortunately there’s no “fix everything” button here and there are tradeoffs everywhere.</p><ul><li><strong>Training data</strong>: (for training a person) ideally you want a low number (1–6) of HD images of the same face with various poses &amp; backgrounds.</li><li><strong>Templates</strong>: We currently have three templates: ‘<strong>person’</strong>, ‘<strong>object</strong>’ and ‘<strong>style</strong>’. ‘person’ is well tested and should give decent results in most situations, the other two are still very experimental, use at your own risk :)</li><li>We highly recommend keeping the defaults, but if you want to adjust the advanced settings please refer to <a href="https://github.com/abraham-ai/lora/">the LORA codebase here</a> if you know what you’re doing.</li></ul><h4>Using a trained LORA model</h4><p>Once your LORA model is trained, there’s a a whole lot of subtle tricks that can make your outputs look amazing if you spend a bit of time tweaking and experimenting.</p><p>First of all, to trigger the learned concept in the prompt, you have to embed the name of the lora model (the text in the dropdown menu) into the prompt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*AE5nqkgVkgYft7-b8tzhyg.png" /><figcaption>When the name of the LORA model is eg “xander” you need to trigger it in the prompt as &lt;xander&gt;.</figcaption></figure><p>Generating images with a LORA using large<em> height </em>x<em> width </em>settings will tend to repeat the learned concept in the image, which is usually not what you want (even though it can sometimes produce funny results). To avoid this, render at lower <em>heigth</em> x <em>width</em> and simply use upscale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TkIUtHTvsf_FfD3PLdhnhA.jpeg" /><figcaption>Rendering with LORA at high resolution without an init_image tends to repeat the learned concept. Prompt: “&lt;xander&gt; taking a selfie in front of the great pyramid of gizeh, Egypt”</figcaption></figure><p>Alternatively, a great way to get really good results with LORA is to use an init_image. This init_image does not have to be high quality as only the rough edges and colors are actually used, but allows you to control the image composition and also render at higher native image resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wcKCxJjC_AEk7x5Luz5HkA.jpeg" /><figcaption>Using a very simple paste / paint init_image (with a low strength of eg 0.15) is a great way to steer the desired image content.</figcaption></figure><p>The final setting to play with is the “LORA scale”. This controls how strongly the trained LORA model is influencing the generation. A scale of 0 will result in the normal base model and will not show the trained concept at all. A scale of 1.0 will fully add the LORA model (which can sometimes be a little bit too much). Ideal values are usually around 0.7–0.9</p><h3>TTS</h3><p>Our text-to-speech endpoint takes a few short input .wav files of a person speaking + a prompt. Our backend will then customize a text-to-speech model using that audio and return a soundfile speaking the prompt with the person’s voice.</p><h3>Wav2Lip</h3><p>Takes an input image + an audio file (eg from TTS) and creates a speaking avatar using lip-synching.</p><h3>Complete</h3><p>Runs autocomplete on an input text using gpt3. (This just calls OpenAI’s API)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b01a2714b15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Train your own GAN]]></title>
            <link>https://xanderst.medium.com/train-your-own-gan-b8b3a20a8836?source=rss-8e7e4128ff60------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8b3a20a8836</guid>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[gans]]></category>
            <category><![CDATA[digital-art]]></category>
            <category><![CDATA[music-visualization]]></category>
            <dc:creator><![CDATA[Arxiv Insights]]></dc:creator>
            <pubDate>Sat, 10 Oct 2020 20:24:40 GMT</pubDate>
            <atom:updated>2021-04-20T11:20:38.404Z</atom:updated>
            <content:encoded><![CDATA[<p>So, you want to train your own GAN-model and create a visual theme that is 100% customized to your needs using your own images? That’s possible, but there are some details to consider, let’s see how it works!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iOBqoUUe6WCEds9oZwQXwA.png" /></figure><p>This blogpost is meant as a guide to Custom Model training for <a href="https://wzrd.ai/"><strong>wzrd.ai</strong></a></p><p>To get a sense of the types of digital audiovisual content that can be created with these models, take a look at:</p><p><a href="https://vimeo.com/neuralsynesthesia">Neural Synesthesia</a></p><h3>Creating videos with GANs</h3><p>The first question people ask us when they want to create their own visual theme is:</p><blockquote><strong><em>“How can I use my own images in a WZRD video?”</em></strong></blockquote><blockquote>The short answer is: <em>you can’t…</em></blockquote><blockquote>The long answer is: <em>it’s tricky, let me explain.</em></blockquote><h3>How GANs work</h3><p>In order to understand why you can’t simply take your own images and use WZRD’s AI to turn those into a beautiful, morphing video; I need to explain you <strong>how GANs work</strong> and how we use them to create music videos.</p><p>A Generative Adversarial Network (GAN) has two main components: <strong>the Generator and the Discriminator. </strong>Both of those are neural networks, but they each have a very specific job to do:</p><ul><li><strong>The Generator</strong> receives a bunch of (randomly chosen) numbers as input and transforms those numbers into an image. Every time it receives different input numbers, it produces a different image, we call this image <strong>‘a fake (or generated) sample’</strong>.</li><li><strong>The Discriminator</strong> receives an image as input and needs to decide if that image is<em> a) a real image from the dataset </em>or<em> </em>b)<em> generated by the Generator. </em>The Discriminator will randomly be shown images from the real images or samples from the generator and needs to learn how to tell those apart.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VgZ9wHdGxHTLM7L0UfwUMA.jpeg" /></figure><p>Now, the Generator’s job is to produce samples that are so realistic that the Discriminator can no longer tell the difference between them and the real images from the dataset, that is the goal of <strong>the training process</strong>.</p><p>Once the GAN is trained, we no longer need the Discriminator, so we throw it away. What we care about is the Generator, since this network can now transform random input numbers into samples that look like images from the dataset: let the magic begin!</p><blockquote>On <a href="https://app.wzrd.ai/">the WZRD dashboard</a>, you can find a bunch of different GAN models that we already trained for you! Just select a visual style, pick your favourite generated images and following the instructions to turn them into a morphing video.</blockquote><h3>The magic</h3><p>The final thing you need to know about GANs is how you can use them to create morphing videos, let’s see how that works!</p><p>Remember that the Generator transforms a bunch of random numbers into an image? Well, if you change those numbers just a tiny, tiny bit; the resulting image will also change a tiny, tiny bit!</p><blockquote><strong>So by slowely changing the input numbers, we can slowely change the generated image, essentially creating a morphing video</strong> (by doing this many times per second).</blockquote><p>When you are picking favourites on the WZRD platform, what you are actually doing is picking sets of input numbers that result in beautiful images when sent through the Generator network.</p><p>Once you click “render video” on the timeline, our AI is going to slowely change those input numbers from the first favourite, to the next and so on.. creating a morphing video that slowely blends from one favourite image to the next.</p><h3>Why you can’t directly use your own images</h3><p>If we are slowely changing input numbers to create a morphing video, you might already see the problem in blending between custom images that you could potentially provide yourself: we don’t know what input numbers we must feed to the Generator in order to produce those images.. And if we don’t know those input numbers, <strong>it is impossible to create a morphing video that blends between those given, real images</strong>.</p><blockquote>In other words, <strong>a GAN can only create videos that morph between images it has generated for itself, not between any image that we give it!</strong></blockquote><h3>So then, how can we create custom visual themes?</h3><p>Luckily, there is a way to create custom visual styles, and that is by uploading loads of real images and <strong>having the GAN learn to reproduce that visual theme</strong>. Unfortunately, the samples that the Generator will learn to produce will not be identical to the images you have uploaded, but, if the training went well, the generated samples should look <strong>similar</strong> to the training images.</p><p>We can then pick our favourites from the samples that the GAN produces and create a video that smoothly morphs between them!</p><p>Below, you can see an example of some real training images (left) and some generated samples (right) from the GAN after training on those images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oA7WLTw49-i292ZwkNetuA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZ2hrNsIlOYm-0p7Vgj2PA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tfhhacgCCyemATQpazthLw.jpeg" /><figcaption>Real images (left) vs Generated samples (right) from WZRD’s infinity model</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F394670850%3Fapp_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F394670850&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F868584248_1280.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1440" height="1440" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ac04c682530efb9d6b5c0844908be81/href">https://medium.com/media/9ac04c682530efb9d6b5c0844908be81/href</a></iframe><p><em>If you’d like to dive deeper into the rabbit hole, you can take a look at </em><a href="https://www.youtube.com/watch?v=dCKbRCUyop8"><em>my video here</em></a><em> where I explain GANs in even more detail, but don’t feel compelled: you know more than enough to continue right here!</em></p><h3>Allright, I’m starting to get it…</h3><blockquote>So, what do I need to create my own, custom GAN model?</blockquote><h3>Training Images</h3><p>Machine Learning models are known to be data hungry and the same applies to GANs. Traditionally, you would need around 50&#39;000 diverse images to train a good generative model, and yes, I know, that is A LOT…</p><p>Luckily, using the latest tricks <em>(Academical references: </em><a href="https://arxiv.org/pdf/2006.10738.pdf"><em>paper1</em></a><em>, </em><a href="https://arxiv.org/pdf/2006.06676.pdf"><em>paper2</em></a><em>)</em>, it is now possible to get almost the same results with much less example images. To be specific, we’re now able to train custom models with as little as ~1000 images. There are a few caveats to keep in mind though:</p><ul><li><strong>Data diversity ⟶ Model diversity</strong>, in other words: the more training images you can provide, the more diverse the samples will be that the GAN is able to generate. Small image sets tend to produce models that exhibit low visual change in their outputs.</li><li>On the other hand, a lot also depends on the inherent diversity of the dataset itself. If all your images are rather similar (eg faces) the model will have an easier job modelling those and generating new, realistic samples can be done with a fairly limited set of training images. For visual styles that have loads of diversity (say landscapes for example), the model has a much harder job to do and more images are usually needed to get good results.</li></ul><p>As <strong>a rule of thumb:</strong></p><ul><li>~1000 images: just doable, but the model might lack some diversity</li><li>1000–2000 images: good</li><li>2000–4000 images: great</li><li>4k+ images: purrrfect!</li></ul><p>To quantify these rules of thumb a bit more rigorously, we can look at something called <strong>the “FID” score of a GAN model</strong>. Briefly explained, FID is a statistic that measures the visual difference between the real and generated images of a model. If FID is zero, the generated images are essentially indistinguishable from the real ones, so <strong>lower FID is better</strong>.</p><p>Coupling this metric (vertical axis) with the size of the training data (horizontal axis) you can see why you need as many images as you can get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*nEpWDDQbt6Q8qo9bkiDB4A.jpeg" /><figcaption>More images yields better GAN models: FID scores (vertical axis) of GANs using different amounts of training data (horizontal axis) and different training methods (colors). <strong>WZRD’s backend uses the orange method</strong> to train custom models.</figcaption></figure><h3>Computational Resources</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksnA_iLgbqijaEeq4DZxwQ.jpeg" /></figure><p>Even though a lot of progress has been made, GANs are still very compute-hungry to train. One of the reasons is that GANs essentially involve two neural networks (the Generator and the Discriminator) that both have to be trained simultaneously.</p><p>As such, <strong>training a good GAN usually requires about 100–200 hours of training time on a single GPU</strong>. It goes without saying that training a model for longer will improve results, so there is a cost-quality trade-off here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*yEOeWLhzZPB8kyH5M8c1YA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t2C5n1DnqAMISsP1rOz1CA.jpeg" /><figcaption>Left: Model performance vs Training time (<a href="https://github.com/NVlabs/stylegan2-ada">source</a>) | Right: Model performance vs Training steps and Dataset size (colors) (<a href="https://arxiv.org/pdf/2006.06676.pdf">source</a>)</figcaption></figure><h3>Cost</h3><p>Because of the required computational resources, training custom GAN models is expensive. If you look at the GPU pricing table <a href="https://cloud.google.com/compute/gpus-pricing">here</a> and consider that the above charts are using V100 GPU’s, you can understand why <strong>training a custom model will set you back at least $500.</strong></p><h3>Sample quality</h3><p>Finally, while GANs have seen impressive progress over the past 5 years, they cannot just model any kind of image dataset, <strong>there are limits</strong> to what these AI models can currently do.</p><p>It is a good idea to try and manually curate the image dataset so that it contains only “<strong>good</strong>” images which means:</p><ul><li>High definition (&gt; 1000 x 1000 pixels per image)</li><li>The primary object of focus is clear &amp; centered in the image</li><li>The dataset is not too diverse, as this will simply result in what is called “Mode dropping” where the GAN is not able to model all the diversity of the dataset and simply drops parts of the visual theme from it’s output.</li></ul><h3>Examples</h3><p>To give an idea of how the WZRD platform can be used, here are some sample images from the training data + the generated samples from several models publically available on the WZRD platform. For every model, a video example is also included that shows how the model can be used to create various styles of digital audio-reactive content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRPgn7MVlQ3Xs_PMv_ZEjA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BjIs9nL_ChixTmcKeG1vUg.jpeg" /><figcaption>Real images (left) vs Generated samples (right) of WZRD’s Religion model, trained for 10 days on 15k images.</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F448593420%3Fapp_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F448593420&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F942263081_640.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="720" height="900" frameborder="0" scrolling="no"><a href="https://medium.com/media/5978fbe0c93aaace9fd150f47ddf70bc/href">https://medium.com/media/5978fbe0c93aaace9fd150f47ddf70bc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hMxpDNWlxWnmqIGCU4mZWw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gynAWYFIHrAjTlcEIqeatw.jpeg" /><figcaption>Real images (left) vs Generated samples (right) of WZRD’s Earth model, trained for 5 days on 5k images.</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FN6RWp611yUk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DN6RWp611yUk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FN6RWp611yUk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a3f5bfb7dce902685120b999e814ef4/href">https://medium.com/media/4a3f5bfb7dce902685120b999e814ef4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebuOYDndNOgnP0elYBe-VQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kVOLR_apuELlyMkVXffAbQ.jpeg" /><figcaption>Real images (left) vs Generated samples (right) of WZRD’s Dots model, based on the amazing artworks of <a href="https://www.nojbarker.com/">Noj Barker</a>, trained for 4 days on 1k images.</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F466938117%3Fapp_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=http%3A%2F%2Fvimeo.com%2F466938117&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F972927004_640.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="720" height="720" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e67ad9f6defbad56ab7d4ff3bb03df2/href">https://medium.com/media/6e67ad9f6defbad56ab7d4ff3bb03df2/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8b3a20a8836" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>