<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Christopher Landschoot on Medium]]></title>
        <description><![CDATA[Stories by Christopher Landschoot on Medium]]></description>
        <link>https://medium.com/@crlandschoot?source=rss-b64548f914a5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*q0SJqZRuuuY4mliJwhiHKg@2x.jpeg</url>
            <title>Stories by Christopher Landschoot on Medium</title>
            <link>https://medium.com/@crlandschoot?source=rss-b64548f914a5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:55:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@crlandschoot/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The State of Generative Music]]></title>
            <link>https://medium.com/whitebalance/the-state-of-generative-music-0fcb2745baf9?source=rss-b64548f914a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/0fcb2745baf9</guid>
            <category><![CDATA[audio]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-music]]></category>
            <dc:creator><![CDATA[Christopher Landschoot]]></dc:creator>
            <pubDate>Wed, 27 Nov 2024 15:19:31 GMT</pubDate>
            <atom:updated>2024-12-02T15:34:01.346Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A seamless visual progression from vintage analog music technology to futuristic AI-generated soundscapes, unified by a continuous thread of musical elements and evolving tones, symbolizing the evolution of music innovation." src="https://cdn-images-1.medium.com/max/1024/1*C6XGoUj2QYuVvhym8q_kzw.jpeg" /><figcaption>Image Generated with <a href="https://blackforestlabs.ai/">Flux</a></figcaption></figure><h4>Understanding the Impacts of New Generative Music Technologies on <em>Music, Artists, and Rights Holders</em></h4><p>This year has been a whirlwind for generative music and audio AI. The idea of generative music has <a href="https://en.wikipedia.org/wiki/Generative_music">been around for decades</a>, but only within this past year have we seen it take off from basic MIDI and sample creation to generating full-length, coherent songs with lyrics. It was little more than a year ago I was generating unique, garbled samples with <a href="https://colab.research.google.com/github/harmonai-org/sample-generator/blob/main/Dance_Diffusion.ipynb">Dance Diffusion</a> (from <a href="https://www.harmonai.org/">Harmonai</a>) to use in a <a href="https://soundcloud.com/after-august/viridian">song</a> that I was working on. Admittedly it was very exciting at the time to be able to generate new sounds from scratch. I was so interested in this technology that I put together an open-source project, <a href="https://github.com/crlandsc/tiny-audio-diffusion">tiny-audio-diffusion</a>, geared towards hardware-constrained individuals (like myself) who wanted to get their toes wet training generative audio models. While this felt cutting-edge at the time (my models could only generate 1.5-second one-shot drum samples), here we are just over a year later and commercial generative music models can produce high-quality, full-length songs with lyrics from just a text description.</p><p>With any transformative new technology comes some growing pains, however. These new capabilities have raised alarm bells in the minds of many musicians, producers, rights holders, and creators, that generative music models could be an existential threat to their art and livelihoods. This has generated (pun intended) a rift between AI supporters and detractors, leaving many musicians and rights holders reticent to allow any AI model, generative or not, to use their data during training. This is an unfortunate binarization of an issue that actually has many shades of gray, as AI use cases can range from predatory to neutral to significantly benefiting creators and rights holders.</p><p>To fully comprehend the contentious state of generative AI and explore a potential path forward, it is important to understand the arguments from both sides along with the context in which these disagreements emerged. A shared foundation is essential for fostering more nuanced and productive conversations about AI and its role in music and art.</p><h3>The Difference Between “Generative” and “Non-Generative” Models and Why It Matters</h3><p>As I briefly alluded to earlier, there has been a recent boom in generative music technology which has complicated public perception of the use of AI in audio. Additionally, many companies have begun using Artificial Intelligence (AI), Machine Learning (ML), Generative AI (gen AI), etc. labels interchangeably in an attempt to capitalize on the clout of these buzzwords. Unfortunately, this only leads to more confusion, even though each of these terms has a specific definition. For example, Artificial Intelligence (AI) is an umbrella category encompassing technologies designed to simulate human intelligence, while Machine Learning (ML) is a subset of AI that enables systems to learn and improve from data patterns without explicit programming. More relevantly, much of the tension surrounding AI in music arises from misunderstandings about the differences between <em>Generative</em> and <em>Non-Generative</em> AI.</p><p><strong>Generative AI</strong> can be qualified as <em>a machine learning system that has the ability to create something new, i.e. generate something</em>. Some examples of generative AI in audio are:</p><ul><li><strong>Song or Audio Creation</strong> — e.g. Generating a song from a text prompt</li><li><strong>Symbolic Melody Creation</strong> — e.g. Creating a melody in <a href="https://en.wikipedia.org/wiki/MIDI#:~:text=The%20Standard%20MIDI%20File%20(SMF,mid%20extension.">MIDI format</a>, <a href="https://en.wikipedia.org/wiki/Musical_notation">standard notation</a>, or <a href="https://en.wikipedia.org/wiki/Tablature">tablature</a></li><li><strong>Voice Creation or Cloning</strong> — e.g. Synthesizing a new voice or mimicking a person’s voice</li><li><strong>Song Extension and Inpainting</strong> — e.g. Completing a partially written song or filling gaps in an audio recording</li><li><strong>Music-to-Music Generation and Style Transfer</strong> — e.g. Transforming a rock song into a jazz arrangement</li><li><strong>Audio Upsampling</strong> — e.g. Enhancing the quality of low-resolution or compressed audio files</li><li><strong>Text-to-Speech</strong> — e.g. Converting written text into spoken audio with natural-sounding voices</li></ul><p><strong>Non-generative AI</strong> can be qualified as <em>a machine learning system that can analyze, categorize, or enhance audio without creating something new</em>. Some examples of non-generative audio are:</p><ul><li><strong>Music Recommendation </strong>— e.g. Spotify recommending songs based on listening history</li><li><strong>Audio Source Separation</strong> — e.g. Isolating vocals or instruments from a track</li><li><strong>Music Transcription</strong> — e.g. Converting music into MIDI or sheet music</li><li><strong>Audio Classification</strong> — e.g. Classifying musical genres or different types of sounds</li><li><strong>Automatic Mixing and Mastering</strong> — e.g. Adjusting audio levels and effects to optimize sound quality for production</li><li><strong>Speech-to-Text </strong>— e.g. Converting spoken audio into text</li></ul><p>Because there is a clear distinction between these two categories, they should be addressed as separate entities when discussing music and AI in both general and legal contexts. Many non-generative (and a few generative) machine learning models have actually been around for many years, such as music recommendation systems, and the legality and morality of their uses historically was not a primary focus. Most artists felt that those use cases were generally of benefit to them and did not pose an existential threat to their work. It is only the recent generative music developments that initiated a global rethinking of data rights across the entire gamut of AI in audio.</p><p>While it is important to comprehend this gen vs non-gen distinction, alone it does not offer enough depth to make a case for or against generative audio models, which is where the controversy is currently swirling. It still leaves the question open of whether these models are genuinely creating something new and original. In order to make informed judgments, it is imperative to understand the fundamental principles of how generative music models actually work.</p><h3>How Do Generative Audio Models Work?</h3><p>Many of the state-of-the-art music generation models leverage a process known as <em>diffusion</em>, made famous by the image generation model, <a href="https://stability.ai/news/stable-diffusion-public-release">Stable Diffusion</a>, in the summer of 2022. While not all generative models work in this exact manner, its core concepts are transferable. The scope of this piece does not cover any technical details regarding diffusion models, but if you are interested in learning more about the nuts and bolts, see <a href="https://towardsdatascience.com/audio-diffusion-generative-musics-secret-sauce-f625d0aca800">this article</a>.</p><figure><img alt="Image generated by Stable Diffusion" src="https://cdn-images-1.medium.com/max/929/0*WxQSwSRo56Yp2qs3" /><figcaption>Image generated with Stable Diffusion — <a href="https://stability.ai/news/stable-diffusion-public-release">original Stable Diffusion public release</a></figcaption></figure><p>In the context of image and audio generation, <strong>diffusion</strong> is <em>the process of refining noise into something recognizable</em>. This can be thought of as taking static from an old TV and slowly rearranging the pixels into an image.</p><figure><img alt="Diffusion Process for Image Generation" src="https://cdn-images-1.medium.com/max/1024/1*OsHXjv11DD1MWfcCj9gQpA.jpeg" /><figcaption>Diffusion Process for Image Generation (Image generated with Stable Diffusion)</figcaption></figure><p>The same concept can be applied to audio, but instead of resolving into an image, the model refines the noise into a waveform (in this case music). To teach a model this capability, it is shown millions of hours of music, repeating the noise to music process for each sample. Over time, the model becomes adept at learning how to transform noise into music.</p><figure><img alt="Diffusion Process for Audio Generation" src="https://cdn-images-1.medium.com/max/1024/1*hTy41VBvYjDYJ58UUk9E-w.png" /><figcaption>Diffusion Process for Audio Generation</figcaption></figure><p>The key takeaway from this process is that a generative music model doesn’t learn how to precisely reconstruct the exact music that it has been trained on. This is because the music training data falls within a confined <em>distribution</em> on the scale of all possible sounds. The model learns to generate within the general range of this musical <em>distribution,</em> not to any single point of training data (i.e. a certain song). Therefore, these models can generate “new” music, but it will only fall within the <em>distribution</em> that it has been trained on.</p><p>Think of it like a child that learns how to build with Legos. If the child is shown the instructions for many Lego starship sets, she will be able to build new creations that are similar to the starship sets that she learned from. However, if she was never shown instructions for the Eiffel Tower set, she wouldn’t know how to build anything similar to the Eiffel Tower. So while she is able to build “new” creations, they will all be similar to starships. In the same vein, a model that is trained only on music would not know how to generate the sound of a dog barking.</p><figure><img alt="Young girl building a starship out of legos" src="https://cdn-images-1.medium.com/max/1024/1*MI4PAFp-5iDI8mme-EY9gg.jpeg" /><figcaption>Image generated with <a href="https://blackforestlabs.ai/">Flux</a></figcaption></figure><p>The idea follows that if the scale of the data is large and diverse enough, the model will learn to generalize across all types of music and be able to generate “new” pieces of music within this sufficiently wide distribution. While these explanations are oversimplifications, they illustrate the range of creative capabilities these models possess, which has significant implications for how their outputs should be treated. However, the question of what qualifies as an “original” work is still hotly debated.</p><h3>The Battle of AI Training Data</h3><p>This open question leads to the crux of the matter, what considerations should be taken into account surrounding rights holders’ data and model training. Having recently attended the ISMIR (<a href="https://ismir.net/">International Society for Music Information Retrieval</a>) <a href="https://ismir2024.ismir.net/">conference</a> in San Francisco, I was able to observe first-hand the sentiment within the technical community surrounding this issue. Coupled with my experience as a musician and producer, I have seen diverse opinions from all sides.</p><figure><img alt="Cartoon of 2 groups of people arguing. One group holds a sign reading “Music For Everyone!!” and the other group holds a sign reading “Respect Artist Rights!”" src="https://cdn-images-1.medium.com/max/1024/1*LMCWl1_gbw4ziXaxEpguLA.jpeg" /><figcaption>Image generated with <a href="https://blackforestlabs.ai/">Flux</a></figcaption></figure><p>The parties involved are largely broken down into two camps:</p><p>The first camp’s stance is <strong>“Generative models are creating something original so they should be allowed to train on copyrighted material.”</strong> This group largely consists of practitioners working at AI generation companies. Their arguments are:</p><ul><li>Humans listen to and learn from a wide array of music over the course of their lives which influences their musical creations. Models are no different, as they are influenced by music that they’ve trained on, but don’t replicate it.</li><li>Music creation has always included practices that blur the lines of originality, such as remixing, sampling, and reinterpretation, which are widely accepted in artistic culture. Generative models are seen as a natural extension of these practices.</li><li>Relying on U.S.’s <a href="https://en.wikipedia.org/wiki/Fair_use"><em>Fair Use</em> doctrine</a>, which is generally more permissive than copyright laws in many <a href="https://en.wikipedia.org/wiki/Copyright_law_of_the_European_Union">other countries</a>, they argue that the outputs from generative models are original and <em>transformative,</em> therefore they do not incur damages on the works on which they have been trained.</li><li>Generative AI expands creative possibilities by enabling anyone, regardless of skill level, to create music, thus democratizing music production and lowering barriers to entry.</li><li>These models allow for new forms of artistic collaboration, where humans and AI can co-create works that would not have been possible otherwise.</li><li>Training on a broad dataset of music ensures that AI models can capture the diversity of musical traditions and genres, fostering innovation rather than limiting it to narrow, predefined styles.</li></ul><p>The second camp’s stance is <strong>“Generative model outputs will compete with the works they are trained on, therefore consent is required to use any artist’s work in training.”</strong> This group consists of the remainder of the technical community as well as most musical artists and producers. Their arguments are:</p><ul><li>Using copyrighted material without consent is exploitative, as it leverages artists’ work to create potentially competing outputs without compensation or acknowledgment.</li><li>While the training process may partially mimic how humans are inspired by a wide array of music, a major difference is in the scale. Generative AI systems can produce thousands of songs a minute whereas human musicians often only release several songs or albums a year.</li><li>Generative models can replicate styles, melodies, or even voices, blurring the line between inspiration and outright copying, which could harm the value of the original works.</li><li>The outputs from generative models may saturate the market with similar-sounding content, reducing the diversity of musical expression and making it harder for original works to stand out.</li><li>Artists and producers argue for more transparency, requiring AI companies to disclose the specific datasets used for training, and to seek permission before including copyrighted works.</li><li>The legal frameworks for intellectual property protection are not yet fully equipped to handle AI-generated content, creating uncertainty and potential long-term harm to creators’ livelihoods.</li></ul><figure><img alt="Fair Use" src="https://cdn-images-1.medium.com/max/1024/1*1-Zc6vU_vBqLH02Au1he8Q.png" /></figure><p>I have found that many of these arguments on the legal side center on the U.S.’s <a href="https://en.wikipedia.org/wiki/Fair_use"><em>Fair Use</em> laws</a>, which can be understood through four main principles.</p><ol><li><strong>The Purpose and Character of the Use:</strong> Considers whether the use is for commercial purposes vs. nonprofit or educational purposes. This also takes into account whether the use is <em>transformative</em>, meaning it adds new meaning, purpose, or value to the original work.</li><li><strong>The Nature of the Copyrighted Work:</strong> Considers the type of work being used. Creative works (e.g., music, film) have more copyright protections than factual works (e.g., data compilations).</li><li><strong>The Amount or Substantiality of the Portion Used:</strong> Considers both the quantity and quality of the content used. For music, this could be the difference in training on only small chunks of a song vs. a full song.</li><li><strong>The Effect of the Use on the Potential Market for or Value of the Work: </strong>Considers whether the new use serves as a substitute for the original work, reducing its market value or potential licensing opportunities. In some cases, showing that the use could benefit the original work (e.g., increasing its exposure or audience) might strengthen the fair use argument.</li></ol><p>Both sides have legitimate points, but it is important to note the contention derives from artists feeling exploited by the way the generative AI companies have approached this issue without looping them into the conversation. This creator backlash yields unintended downstream effects for the broader music AI community, as the newly adopted default stance for artists is to resist any AI model (generative or not) training on their data without consent and compensation.</p><p>This marks a paradigm shift from the previous status quo, where earlier non-generative machine learning models could freely train on any publicly accessible data. Much like the widely accepted practice of Google crawling web-data to consolidate search results, music was once commonly used to train ML recommendation, classification, and other analysis systems without much scrutiny. While Google allows websites to opt out of indexing, most choose to allow it due to its perceived net benefit. Rights holders should always be entitled to consent to who uses their content, but it is the reversal of the default position from allowing open access to restricting access to their data that is important to highlight. Imagine if every website online now automatically opted out of indexing unless explicitly opting in.</p><p>None of this is to argue that artists should not have the right to consent to training on their data, be fairly compensated, and remain informed. But it demonstrates that the current path is pinning AI companies and rights holders against each other rather than as collaborators. This leaves the responsibility on the technical music AI community to earn back the trust of artists and creators.</p><p>This may be why the phrase “We want to empower artists, not replace them”, has become somewhat overused and cliché. As the belief that an AI company is working in bad faith has become more common, it has been used as a signal to artists that a company wants to work as a collaborator rather than an adversary. I was heartened to see that the majority of the ISMIR community rallied around this sentiment and is actively researching and developing ways to solve some of these outstanding challenges. Additionally, when I have talked with artists and producers one-on-one, they have expressed openness to working with AI in music provided their concerns are taken into consideration. They simply want uses of their work to remain above board and have the ability to consent to their data being used in training models.</p><p>So while there are no immediate solutions to healing this rift, here is a good place for companies to start:</p><ul><li><strong>Abide By the 4 Factors of Fair Use</strong> — The community and courts still need to determine what constitutes “originality” regarding generative AI. While this is admittedly a very complex subject to quantify, the qualitative answer can actually derive from a simple moral gut check — “if it feels wrong, it probably is”.</li><li><strong>Communication</strong> — Offer transparency about the data that models are being trained on. Present clarity on the type of model (e.g. generative, non-generative) and its intent (e.g. generation, classification). Foster opportunities for collaboration between rights holders and AI companies.</li><li><strong>Consent Management </strong>— Allow rights holders to opt out of using their data to train models on a case by case basis. The default choice to opt-in or opt-out requires further discussion and may depend on the model type and intent.</li><li><strong>Licensing &amp; Attribution</strong> — Work with rights holders to determine an appropriate attribution model for each use case. This could range from one-time licensing for training to revenue-sharing to fractional attribution.</li><li><strong>Education</strong> — Inform artists and rights holders about AI technology so they know what they are signing up for or opting out of.</li><li><strong>Copyright Tracking</strong> — Encourage the development of technical tools that allow artists to track where their work has been used across AI and beyond.</li></ul><p>Of course, each of these points can lead to a bevy of questions and debates. But having a shared knowledge of how these models work, the parties involved, and the state of the industry is a prerequisite to discussing these complex topics in a considerate and nuanced manner. These conversations are essential to solving such pressing issues. No one wants a future devoid of genuine music, art, or creativity, so reducing the emotional responses and finding a common starting place is the best (dare I say <em>only</em>) path forward.</p><p><em>All images, unless otherwise noted, are by the author.</em></p><p>I am an audio machine learning engineer and researcher at <a href="https://www.whitebalance.co/">Whitebalance</a> as well as a lifelong musician.</p><p>Find me on <a href="https://www.linkedin.com/in/christopher-landschoot/">LinkedIn</a> &amp; <a href="https://github.com/crlandsc">GitHub</a> and keep up to date with my current work and research here: <a href="https://www.chrislandschoot.com/">chrislandschoot.com</a></p><p>Find my music on <a href="https://open.spotify.com/artist/2i6noWJnJQPXPsudoiJuMS?si=fvLOxUPqTAWKB894WXMz5Q">Spotify</a>, <a href="https://music.apple.com/us/artist/after-august/259370281">Apple Music</a>, <a href="https://www.youtube.com/AfterAugust">YouTube</a>, <a href="https://soundcloud.com/after-august">SoundCloud</a>, and other streaming platforms as <a href="https://www.instagram.com/the_after_august/">After August</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fcb2745baf9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/whitebalance/the-state-of-generative-music-0fcb2745baf9">The State of Generative Music</a> was originally published in <a href="https://medium.com/whitebalance">Whitebalance</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Audio Diffusion: Generative Music’s Secret Sauce]]></title>
            <link>https://medium.com/data-science/audio-diffusion-generative-musics-secret-sauce-f625d0aca800?source=rss-b64548f914a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f625d0aca800</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[thoughts-and-theory]]></category>
            <category><![CDATA[audio]]></category>
            <category><![CDATA[diffusion]]></category>
            <category><![CDATA[music]]></category>
            <dc:creator><![CDATA[Christopher Landschoot]]></dc:creator>
            <pubDate>Mon, 22 Jan 2024 19:41:52 GMT</pubDate>
            <atom:updated>2024-01-22T19:41:52.020Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Futuristic fusion of music and AI in vibrant detail" src="https://cdn-images-1.medium.com/max/1024/1*N2njSffV9qhHwnQqx_xurA.png" /><figcaption>Image generated with DALL·E</figcaption></figure><h4>Exploring the principles behind diffusion technology and how it is being used to create groundbreaking AI tools for artists and producers.</h4><p>Much has been made of the hype of recent generative music AI algorithms. Some see it as the future of creativity and others see it as the death of music. While I tend to lean towards the former camp, as an engineer and researcher, I generally attempt to view these advancements through a more objective lens. With this in mind, I wanted to offer an introduction to one of the core technologies fueling the world of generative audio and music: <a href="https://en.wikipedia.org/wiki/Diffusion_model"><em>Diffusion</em></a>.</p><p>My goal is not to sell or diminish the hype, but rather shed some light on what is happening under the hood so that musicians, producers, hobbyists, and creators, can better understand these new seemingly magic music-making black boxes. I will answer what it means by the claim that these AI algorithms are “creating something completely new” and how that differs from human originality. I hope that a clearer picture will lower the collective temperature and provide insight into how these powerful technologies can be leveraged to the benefit of the creator.</p><p>This piece will touch on technical topics, but you do not need an engineering background to follow along. Let’s begin with some context and definitions.</p><h4>Background</h4><p>The term “AI-generated” has become pervasive across the music industry, but what qualifies as “AI-generated” is actually quite clouded. Eager to jump on the hype of the buzzword, this claim is casually tossed around whether AI is used to emulate an effect, automatically mix or master, separate <a href="https://en.wikipedia.org/wiki/Stem_(audio)">stems</a>, or augment timbre. As long as the final audio has been touched in some way by AI, the term gets slapped on the entire piece. However, the vast majority of music presently released continues to be primarily generated through human production (yes, even ghostwriter’s “<a href="https://youtu.be/7HZ2ie2ErFI?si=Ie3954lyyj7l-tEG">Heart On My Sleeve</a>” 👻).</p><p>Even though this “AI-generated” term is becoming hackneyed for clicks, an appropriate use is when new sounds truly are being created by a computer, i.e. <em>Generative Audio</em>.</p><p>Audio generation can encompass the creation of sound effects samples, melodies, vocals, and even full songs. The two main ways that this is achieved are via <em>MIDI Generation</em> and A<em>udio Waveform Generation</em>. <a href="https://blog.landr.com/what-is-midi/">MIDI (Musical Instrument Digital Interface)</a> generation has a much lower computational cost and can provide high-quality outputs, as the generated MIDI data is then run through an existing virtual instrument to produce sounds. This is the same concept as a producer programming MIDI on a <a href="https://blog.landr.com/piano-roll/">piano roll</a> and playing it through a <a href="https://en.wikipedia.org/wiki/Virtual_Studio_Technology">VST</a> plugin such as <a href="https://xferrecords.com/products/serum/">Serum</a>.</p><figure><img alt="MIDI Piano Roll in Pro Tools" src="https://cdn-images-1.medium.com/max/1024/1*-Knftczs8icqbEMed_Jb1w.png" /><figcaption>MIDI Piano Roll in Pro Tools</figcaption></figure><p>While this is compelling, it is only partially generative, as no audio is actually produced by AI, just as humans cannot synthesize instrument sounds from thin air. The creative capabilities are further limited by whatever virtual instruments the algorithm has access to. Even with these limitations, products implementing this technique, such as <a href="https://www.aiva.ai/">AIVA</a> and <a href="https://www.lemonaide.ai/">Seeds by Lemonaide</a>, can generate quite compelling outputs.</p><p>Audio waveform generation is a much more complicated task, as it is an end-to-end system that does not rely on any external technology. In other words, it produces sounds from scratch. This process most precisely aligns with the true definition of “AI-generated” audio.</p><p>Audio waveform generation can be accomplished using various approaches and yield different outcomes. It can produce single samples, such as <a href="https://audialab.com/features/">Audialab’s ED 2</a> and <a href="https://audialab.com/humanize/">Humanize</a> or my previous work with <a href="https://towardsdatascience.com/tiny-audio-diffusion-ddc19e90af9b">Tiny Audio Diffusion</a>, all the way up to full songs, with models such as <a href="https://google-research.github.io/seanet/audiolm/examples/">AudioLM</a>, <a href="https://flavioschneider.notion.site/flavioschneider/Audio-Generation-with-Diffusion-c4f29f39048d4f03a23da13078a44cdb">Moûsai</a>, <a href="https://www.riffusion.com/">Riffusion</a>, <a href="https://ai.honu.io/papers/musicgen/">MusicGen</a>, and <a href="https://www.stableaudio.com/">Stable Audio</a>. Among these state-of-the-art models, many leverage some form of <em>Diffusion</em> to generate sounds. You have likely at least tangentially heard of diffusion from <a href="https://stability.ai/stable-diffusion">Stable Diffusion</a> or any of the other top-performing image generation models that took the world by storm. This generative method can be applied to audio as well. But what does any of it actually mean?</p><h3>What is Diffusion?</h3><h4>The Basics</h4><p>In the context of AI, diffusion simply refers to the process of adding or removing noise from a signal (like static from an old TV). <em>Forward Diffusion</em> adds noise to a signal (<em>Noising</em>) and <em>Reverse Diffusion</em> removes noise (<em>Denoising</em>). At a conceptual level, diffusion models take white noise and step through the denoising processes until the audio resembles something recognizable, such as a sample or a song. This process of denoising a signal is the secret sauce in the creativity of many generative audio models.</p><figure><img alt="Animation of Audio Waveform Diffusion" src="https://cdn-images-1.medium.com/max/432/0*7OtpaNR2hvOicvAL" /><figcaption>Audio Waveform Diffusion (Source: <a href="https://github.com/simonrouard/CRASH">CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis (Rouard, Hadjeres)</a>)</figcaption></figure><p>This process was originally developed for images. Looking at how the noise resolves into an image (for instance, a puppy sitting next to a tennis ball) provides an even clearer example of how these models work.</p><figure><img alt="Stable diffusion process using an image of a puppy as an example" src="https://cdn-images-1.medium.com/max/1024/0*ekKyxqQ68uTRH_Vp" /><figcaption>Image Diffusion (Image generated with Stable Diffusion)</figcaption></figure><p>With a conceptual understanding, let’s take a look under the hood at the key components of the architecture of an audio diffusion model. While this will veer into the technical side of things, stick with me, as a deeper understanding of how these algorithms work will better illustrate why and how they produce the results that they do (if not, you can always just ask <a href="https://chat.openai.com/">ChatGPT</a> for the Cliff Notes).</p><h4>The U-Net Model Architecture, Compression, and Reconstruction</h4><p>At the core of an audio diffusion model is the <a href="https://arxiv.org/abs/1505.04597"><em>U-Net</em></a>. Originally developed for medical image segmentation and aptly named for its resemblance to a U, the U-Net has been adapted to generative audio due to its powerful ability to capture both local and global features in data. The original U-Net was a 2-dimensional <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network"><em>convolutional neural network</em></a> (CNN) used for images, but can be adapted to 1-dimensional convolution to work with audio waveform data. See a visual representation of the original U-Net architecture (for images) below.</p><figure><img alt="U-net framework" src="https://cdn-images-1.medium.com/max/1024/1*OAHceZXifK2Yi3W-px5NmQ.png" /><figcaption>U-Net (Source: <a href="https://arxiv.org/abs/1505.04597v1">U-Net: Convolutional Networks for Biomedical Image Segmentation (Ronneberger, et. al)</a>)</figcaption></figure><p>Similar to a <a href="https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73">Variational Autoencoder (VAE)</a>, A U-Net consists of an <em>encoder</em> (the left side of the U) and a <em>decoder</em> (the right side of the U), connected by a <em>bottleneck</em> (the bottom layer of the U). Unlike a VAE, however, a U-Net hosts <em>skip connections</em> (shown by the horizontal gray arrows) that link the encoder to the decoder which is the key piece to producing high-resolution outputs. The encoder is responsible for capturing the <a href="https://en.wikipedia.org/wiki/Feature_(machine_learning)"><em>features</em></a>, or characteristics, of the input audio signal while the decoder is responsible for the reconstruction of the signal.</p><p>To aid the visualization, picture the audio data entering the top left side of the U, following the red and blue arrows down the encoder to the bottleneck at the bottom, and then back up the decoder following the blue and green arrows to the top right of the U. Each blue rectangle represents a model <em>layer</em>. At each level of the encoder, the input audio signal is compressed further and further until it reaches a highly condensed representation of the sound at the base of the U (the bottleneck). The decoder then takes this compressed signal and effectively reverses the process to reconstruct the signal. Each layer (blue rectangle) the data passes through has a series of adjustable weights associated with it that can be thought of as millions of tiny knobs that can be turned to tweak this compression/reconstruction process. Having layers at different levels of compression allows the model to learn a range of features from the data, from large-scale features (e.g. melody and rhythm) to fine-grained details (e.g. high-frequency timbral characteristics).</p><p>Using an analogy, you can think of this full system as the process required to create an <a href="https://en.wikipedia.org/wiki/MP3">MP3</a> audio file and then listen to that MP3 on a playback device. At its core, an MP3 is a compressed version of an audio signal. Imagine that the encoder’s job is to create a new type of compressed audio format, just like an MP3, in order to consistently condense an audio signal as much as possible without losing fidelity. Then the decoder’s job is to act like your iPhone (or any playback device) and reconstruct the MP3 into a high-fidelity audio representation that can be played through your headphones. The bottleneck can be thought of as this newly created MP3-type format, itself. <strong>The U-Net represents the process of compression and reconstruction, not the audio data</strong>. The model can then be trained with the goal of being able to accurately compress and reconstruct a wide range of audio signals.</p><p>This is all well and good, but we haven’t generated anything yet. We’ve only constructed a way to compress and reconstruct an audio signal. However, this is the base process required to begin generating new audio, and it only requires a slight adjustment to do so.</p><h4>Noising and Denoising</h4><p>Let&#39;s revisit the idea of <em>noising</em> and <em>denoising</em> that we touched on earlier. Theoretically, we had some magic model that could be taught to take some white noise and “denoise” it into recognizable audio, perhaps a beautiful concerto. A critical requirement of this magic model is that it must be able to reconstruct the input audio signal at high fidelity. Luckily, the U-Net architecture is designed to do exactly that. So the next piece of the puzzle is to modify the U-Net to perform this denoising process.</p><p>Counter-intuitively, to teach a model to denoise an audio signal, it is first taught how to add noise to a signal. Once it has learned this process, it inherently knows how to perform the inverse in order to denoise a signal.</p><p>Recall in the previous section that we detailed how the U-Net can learn to compress and reconstruct an audio signal. The noising process follows nearly the same formula, but instead of reconstructing the exact input audio signal, the U-Net is directed to reconstruct the input audio signal with a small amount of noise added to it. This can be visualized as reversing the steps taken in the earlier series of images of the puppy.</p><figure><img alt="Diffusion noising steps using an image of a puppy as an example" src="https://cdn-images-1.medium.com/max/1024/1*dny2lUi1ACTwrIFTR5oeBg.png" /><figcaption>Diffusion Noising Steps (Image generated with Stable Diffusion)</figcaption></figure><p>The process of adding noise to a signal must be probabilistic (i.e. predictable). The model is shown an audio signal and then instructed to predict the same signal with a small amount of <a href="https://en.wikipedia.org/wiki/Gaussian_noise#:~:text=In%20signal%20processing%20theory%2C%20Gaussian,can%20take%20are%20Gaussian%2Ddistributed.">Gaussian noise</a> added to it. Because of its properties, Gaussian noise is most commonly used but it is not required. The noise must be defined by probabilistic distribution, meaning that it follows a specific pattern that is consistently predictable. This process of instructing the model to add small amounts of predictable noise to the audio signal is repeated for a number of <em>steps</em> until the signal has effectively become just noise.</p><figure><img alt="Animation of Reverse Audio Waveform Diffusion" src="https://cdn-images-1.medium.com/max/432/1*ZCdCK_FcI_9ZgicKhBiPbg.gif" /><figcaption>Noising a Snare Sample (Source: <a href="https://github.com/simonrouard/CRASH">CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis (Rouard, Hadjeres)</a>)</figcaption></figure><p>For example, let’s take a one-shot sample of a snare drum. The U-Net is provided this snare sample and it is asked to reconstruct that snare sound, but with a little noise added making it sound a little less clean. Then this slightly noisy snare sample is provided to the model, and it is again instructed to reconstruct this snare sample with even more noise. This cycle is repeated until it sounds as if the snare sample no longer exists, rather only white noise remains. The model is then taught how to do this for a wide range of sounds. Once it becomes an expert at predicting how to add noise to an input audio signal, because the process is probabilistic, it can simply be reversed so that at each step a little noise is removed. This is how the model can generate a snare sample when provided with white noise.</p><p><strong>Because of the probabilistic nature of this process, some incredible capabilities arise, specifically the ability to simulate creativity.</strong></p><p>Let’s continue with our snare example. Imagine the model was trained on thousands of one-shot snare samples. You would imagine that it could take some white noise and then turn it into any one of these snare samples. However, that is not exactly how the model learns. Because it is shown such a wide range of sounds, it instead learns to create sounds that are generally similar to any of the snares that it has been trained on, but not exactly. This is how brand new sounds are created and these models appear to exhibit a spark of creativity.</p><p>To illustrate this, let’s use the following sketch.</p><figure><img alt="Sketch of conceptual sound space" src="https://cdn-images-1.medium.com/max/1024/1*L8naco7ZFOvkdVJ51WyEHg.png" /></figure><p>Pretend that all possible sounds, from guitar strums to dog barks to white noise, can be plotted on a 2-dimensional plane represented by the black rectangle in the image above. Within this space, there is a region where snare hits exist. They are somewhat grouped together because of their similar timbral and transient characteristics. This is shown by the blue blob and each blue dot is representative of a single snare sample that we trained our model on. The red dots represent the fully noised versions of the snares the model was trained on and correspond to their un-noised blue dot counterparts.</p><p>In essence, our model learned to take dots from the “not snare” region and bring them into the “snare” region. So if we take a new green dot in the “not snare” region (e.g. random noise) that does not correspond to any blue dot, and ask our model to bring it into the “snare” region, it will bring it to a new location within that “snare” region. This is the model generating a “new” snare sample that contains some similarities to all other snares it was trained on in the snare region, but also some new unknown characteristics.</p><p>This concept can be applied to any type of sound, including full songs. This is an incredible innovation that can lead to numerous new ways to create. It is important to understand that these models will not create something outside of the bounds of how they are trained, however. As shown in the previous illustration, while our conceptual model can take in any type of sound, it can only produce snare samples similar to those it was trained on. This holds true for any of these audio diffusion models. Because of this, it is critical to train models on extensive datasets so the known regions (like the snare region) are sufficiently diverse and large enough to not simply copy the training data.</p><p><strong>All of this means that no model can replicate human creativity, just simulate variations of it.</strong></p><h4>Applications of Diffusion Models</h4><p>These models will not magically generate new genres or explore unknown sonic landscapes as humans do. With this understanding, these generative models should not be viewed as a replacement for human creativity, but rather as tools that can enhance creativity. Below are just a few ways that this technology can be leveraged for creative means:</p><ul><li><strong>Creativity Through Curation</strong>: Searching through sample packs to find a desired sound is a common practice in production. These models can effectively be used as a version of an “unlimited sample pack”, enhancing an artist’s creativity through the curation of sounds.</li><li><strong>Voice Transfer</strong>: Just like how diffusion models can take random noise and change it into recognizable audio, they can also be fed other sounds and “transfer” them to another type of sound. If we take our previous snare model, for example, and feed it a kick drum sample instead of white noise, it will take the kick sample and begin to morph it into a snare sound. This allows for very unique creations, being able to combine the characteristics of multiple different sounds.</li><li><strong>Sound Variability (Humanization)</strong>: When humans play a live instrument, such as a hi-hat on a drum set, there is always inherent variability in each hit. Various virtual instruments have attempted to simulate this via a number of different methods, but can still sound artificial and lack character. Audio diffusion allows for the unlimited variation of a single sound, which can add a human element to an audio sample. For example, if you program a drum kit, audio diffusion can be leveraged so that each hit is slightly different in timbre, velocity, attack, etc. to humanize what might sound like a stale performance.</li><li><strong>Sound Design Adjustments</strong>: Similar to the human variability potential, this concept can also be applied to sound design to create slight changes to a sound. Perhaps you mostly like the sound of a door slam sample, but you wish that it had more body or crunch. A diffusion model can take this sample and slightly change it to maintain most of its characteristics while taking on a few new ones. This can add, remove, or change the spectral content of a sound at a more fundamental level than applying an EQ or filter.</li><li><strong>Melody Generation:</strong> Similar to surfing through sample packs, audio diffusion models can generate melodies that can spark ideas to build on.</li><li><strong>Stereo Effect</strong>: There are several different mixing tricks to add stereo width to a single-channel (mono) sound. However, they can often add undesired coloration, delay, or phase shifts. Audio diffusion can be leveraged to generate a sound nearly identical to the mono sound, but different enough in its content to expand the stereo width while avoiding many of the unwanted phenomena.</li><li><strong>Super Resolution:</strong> Audio diffusion models can enhance the resolution and quality of audio recordings, making them clearer and more detailed. This can be particularly useful in audio restoration or when working with low-quality recordings.</li><li><strong>Inpainting:</strong> Diffusion models can be leveraged to fill in missing or corrupted parts of audio signals, restoring them to their original or improved state. This is valuable for repairing damaged audio recordings, completing sections of audio that may be missing, or adding transitions between audio clips.</li></ul><h3>Conclusion</h3><p>There is no doubt that these new generative AI models are incredible technological advancements, independent of whether they are viewed in a positive or negative light. There are many more aspects to diffusion models that can optimize their performance regarding speed, diversity, and quality, but we have discussed the base principles that govern the functionality of these models. This knowledge provides a deeper context into what it really means when these models are generating “new sounds”.</p><p>On a broader level, it is not only the music, itself, that people care about — it is the human element in the creation of that music. Ask yourself, if you were to hear a recording of a virtuosic lightning-fast guitar solo, would you be impressed? It all depends. If it was artificially generated by a virtual MIDI instrument programmed by a producer, you will likely be unphased and may not even like how it sounds. However, if you know an actual guitarist played the solo on a real guitar, or even saw him or her do it, you will be completely enamored by their expertise and precision. We are drawn to the deftness in a performance, the thoughts and emotions behind lyrics, and the considerations that go into each decision when crafting a song.</p><p>While these incredible advancements have led to some existential dread for artists and producers, AI can never take that human element away from the sounds and music that we create. So we should approach these new advancements with the intent that they are tools for enhancing artists’ creativity rather than replacing it.</p><p><em>All images, unless otherwise noted, are by the author.</em></p><p>I am an audio machine learning engineer and researcher as well as a lifelong musician. If you are interested in more audio AI applications, see my previously published articles on <a href="https://medium.com/towards-data-science/tiny-audio-diffusion-ddc19e90af9b">Tiny Audio Diffusion</a> and <a href="https://medium.com/rock-nheavy/the-music-demixing-ai-revolution-9d1528c6ef7c">Music Demixing</a>.</p><p>Find me on <a href="https://www.linkedin.com/in/christopher-landschoot/">LinkedIn</a> &amp; <a href="https://github.com/crlandsc">GitHub</a> and keep up to date with my current work and research here: <a href="https://www.chrislandschoot.com/">www.chrislandschoot.com</a></p><p>Find my music on <a href="https://open.spotify.com/artist/2i6noWJnJQPXPsudoiJuMS?si=fvLOxUPqTAWKB894WXMz5Q">Spotify</a>, <a href="https://music.apple.com/us/artist/after-august/259370281">Apple Music</a>, <a href="https://www.youtube.com/AfterAugust">YouTube</a>, <a href="https://soundcloud.com/after-august">SoundCloud</a>, and other streaming platforms as <a href="https://www.instagram.com/the_after_august/">After August</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f625d0aca800" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/audio-diffusion-generative-musics-secret-sauce-f625d0aca800">Audio Diffusion: Generative Music’s Secret Sauce</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tiny Audio Diffusion: Waveform Diffusion That Doesn’t Require Cloud Computing]]></title>
            <link>https://medium.com/data-science/tiny-audio-diffusion-ddc19e90af9b?source=rss-b64548f914a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddc19e90af9b</guid>
            <category><![CDATA[music]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <category><![CDATA[audio]]></category>
            <category><![CDATA[editors-pick]]></category>
            <dc:creator><![CDATA[Christopher Landschoot]]></dc:creator>
            <pubDate>Thu, 29 Jun 2023 19:10:38 GMT</pubDate>
            <atom:updated>2023-06-29T19:10:38.414Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Astronaut Playing A Guitar In Space" src="https://cdn-images-1.medium.com/max/1024/1*Eu0qPhppITZ9pwjBv1J5-A.png" /><figcaption>Generated with Stable Diffusion</figcaption></figure><h4>Exploring how to train models and generate sounds with audio waveform diffusion on a consumer laptop and GPU with less than 2GB VRAM</h4><h3>Background</h3><p>Diffusion models are all the rage currently, especially since <a href="https://stability.ai/blog/stable-diffusion-public-release">Stable Diffusion</a> took the world by storm this past summer. Since then, countless variations and new diffusion models have been published in a wide variety of contexts. And while the stunning visuals have stolen the spotlight, there has been significant development with diffusion related to generative audio.</p><p>Fueled by diffusion and other methods, generative music has seen many recent triumphs, as new models are published all the time. OpenAI wowed the world with the capabilities of <a href="https://openai.com/research/jukebox">Jukebox</a> when it was released in 2020. But Google said “Hold my model” when it produced the remarkable <a href="https://google-research.github.io/seanet/musiclm/examples/">MusicLM</a> at the beginning of this year. Meta was not far behind when they released and open-sourced <a href="https://ai.honu.io/papers/musicgen/">MusicGen</a> this past month. But large institutions are not the only ones joining in, as there have been very interesting contributions from independent researchers such as <a href="https://www.riffusion.com/">Riffusion</a> (Forsgren &amp; Martiros) and <a href="https://arxiv.org/abs/2301.11757">Moûsai</a> (Schneider, et al.). On top of these, numerous other models have been released within the last few years, all having their benefits and drawbacks.</p><p>Diffusion models have captivated so many due to their remarkable creative ability; something that many other genres of machine learning (ML) lack. Most ML models are trained to perform a task and their success can be measured by being correct vs incorrect. But when we enter the realm of art and music, how can a model be optimized to what might be considered best? It could of course learn to reproduce famous art or music, but without novelty, there is no point. So how can this problem be solved — to inject creativity into a machine that only knows 1s and 0s? Diffusion is one method that offers an elegant solution to this quandary.</p><h4>Diffusion — From 10,000 feet</h4><p>At its core, diffusion in ML is simply the process of adding or removing noise from a signal (think of static from an old TV). <em>Forward diffusion</em> adds noise to a signal and <em>reverse diffusion</em> removes noise. The process we are most familiar with is the reverse diffusion process, where the model takes in noise and then “denoises” it into something that humans recognize (art, music, speech, etc.). This process can be manipulated in a myriad of ways to serve different purposes.</p><p>The “creativity” in diffusion comes from the random noise that initiates the denoising process. If you are providing the model with a different starting point every time to denoise into some form of art or music, this simulates creativity as the outputs will always be unique.</p><figure><img alt="Diffusion Process" src="https://cdn-images-1.medium.com/max/1024/1*Y_uCqPvx7gM0Nn1vRNSS_w.png" /></figure><figure><img alt="Diffusion Process" src="https://cdn-images-1.medium.com/max/1024/1*mN7-_GguU1D1_MqdfNI-5g.png" /></figure><figure><img alt="Diffusion Process" src="https://cdn-images-1.medium.com/max/1024/1*Mve8Tzn-BWa5Ubcn4ZY_Sg.png" /></figure><figure><img alt="Diffusion Process" src="https://cdn-images-1.medium.com/max/1024/1*pwhtB9lZ4luRhlCZsl_wRw.png" /><figcaption>Images Generated with Stable Diffusion</figcaption></figure><p>The method of teaching a model to perform this denoising process may actually be a bit counter-intuitive from an initial thought. The model actually learns to denoise a signal by doing the exact opposite — adding noise to a clean signal over and over again until only noise remains. The idea is that if the model can learn how to predict the noise added to a signal at each step, then it can also predict the noise removed at each step for the reverse process. The critical element to make this possible is that the noise being added/removed needs to be of a defined probabilistic distribution (typically Gaussian) so that the noising/denoising steps are predictable and repeatable.</p><p>There is far more detail that goes into this process, but this should provide a sound conceptual understanding of what is happening under the hood. If you are interested in learning more about diffusion models (mathematical formulations, scheduling, latent space, etc.), I recommend reading this <a href="https://www.assemblyai.com/blog/diffusion-models-for-machine-learning-introduction/">blog post</a> by AssemblyAI and these papers (<a href="https://arxiv.org/abs/2006.11239">DDPM</a>, <a href="https://arxiv.org/abs/2102.09672">Improving DDPM</a>, <a href="https://arxiv.org/abs/2010.02502">DDIM</a>, <a href="https://arxiv.org/abs/2112.10752">Stable Diffusion</a>).</p><h3>Tiny Audio Diffusion</h3><h4>Understanding Audio for Machine Learning</h4><p>My interest in diffusion stems from the potential that it has shown with generative audio. Traditionally, to train ML algorithms, audio was converted into a spectrogram, which is basically a heatmap of sound energy over time. This was because a spectrogram representation was similar to an image, which computers are exceptional at working with, and it was a significant reduction in data size compared to a raw waveform.</p><figure><img alt="Spectrogram" src="https://cdn-images-1.medium.com/max/1024/1*TqGtT42pldUIAMTmcSTzOA.png" /><figcaption>Example Spectrogram of a Vocalist</figcaption></figure><p>However, with this transformation come some tradeoffs, including a reduction of resolution and a loss of <em>phase </em>information. The phase of an audio signal represents the position of multiple waveforms relative to one another. This can be demonstrated in the difference between a sine and a cosine function. They represent the same exact signal regarding amplitude, the only difference is a 90° (π/2 radians) phase shift between the two. For a more in-depth explanation of phase, check out this <a href="https://www.youtube.com/watch?v=0XdJTipTnjM&amp;t=318s">video</a> by <a href="https://www.youtube.com/@akashmurthy">Akash Murthy</a>.</p><figure><img alt="phase shift of 90° between sin and cos" src="https://cdn-images-1.medium.com/max/1024/1*oqZxXL5_uaiQPE9hbRnItQ.png" /><figcaption>90° phase shift between <em>sin</em> and cos</figcaption></figure><p>Phase is a perpetually challenging concept to grasp, even for those who work in audio, but it plays a critical role in creating the timbral qualities of sound. Suffice it to say that it should not be discarded so easily. Phase information can also technically be represented in spectrogram form (the complex portion of the transform), just like magnitude. However, the result is noisy and visually appears random, making it challenging for a model to learn any useful information from it. Because of this drawback, there has been recent interest in refraining from transforming audio into spectrograms and rather leaving it as a raw waveform to train models. While this brings its own set of challenges, both the amplitude and phase information are contained within the single signal of a waveform, providing a model with a more holistic picture of sound to learn from.</p><figure><img alt="WAveform" src="https://cdn-images-1.medium.com/max/790/1*wQcZ37MuoHJQDYHGFafetQ.png" /><figcaption>Example Waveform of a Vocalist</figcaption></figure><p>This is a key piece of my interest in waveform diffusion, and it has shown promise in yielding high-quality results for generative audio. Waveforms, however, are very dense signals requiring a significant amount of data to represent the range of frequencies humans can hear. For example, the music industry standard sampling rate is 44.1kHz, which means that 44,100 samples are required to represent just 1 second of mono audio. Now double that for stereo playback. Because of this, most waveform diffusion models (that don’t leverage latent diffusion or other compression methods) require high GPU capacity (usually at least 16GB+ VRAM) to store all of the information while being trained.</p><h4>Motivation</h4><p>Many people do not have access to high-powered, high-capacity GPUs, or do not want to pay the fee to rent cloud GPUs for personal projects. Finding myself in this position, but still wanting to explore waveform diffusion models, I decided to develop a waveform diffusion system that could run on my meager local hardware.</p><h4>Hardware Setup</h4><p>I was equipped with an HP Spectre laptop from 2017 with an 8th Gen i7 processor and GeForce MX150 graphics card with 2GB VRAM — not what you would call a powerhouse for training ML models. My goal was to be able to create a model that could train and produce high-quality (44.1kHz) stereo outputs on this system.</p><h3>Model Architecture</h3><p>I leveraged Archinet’s <a href="https://github.com/archinetai/audio-diffusion-pytorch">audio-diffusion-pytorch</a> library to build this model — thank you to <a href="https://github.com/flavioschneider">Flavio Schneider</a> for his help working with this library that he largely built.</p><h4>Attention U-Net</h4><p>The base model architecture consists of a U-Net with attention blocks which is standard for modern diffusion models. A U-Net is a neural network that was originally developed for image (2D) segmentation but has been adapted to audio (1D) for our uses with waveform diffusion. The U-Net architecture gets its name from its U-shaped design.</p><figure><img alt="U-Net" src="https://cdn-images-1.medium.com/max/1024/1*lvXoKMHoPJMKpKK7keZMEA.png" /><figcaption>U-Net (Source: <a href="https://arxiv.org/abs/1505.04597v1"><em>U-Net: Convolutional Networks for Biomedical Image Segmentation (Ronneberger, et. al)</em></a><em>)</em></figcaption></figure><p>Very similar to an autoencoder, consisting of an encoder and a decoder, a U-Net also contains skip connections at each level of the network. These skip connections are direct connections between corresponding layers of the encoder and decoder, facilitating the transfer of fine-grained details from the encoder to the decoder. The encoder is responsible for capturing the important features of the input signal, while the decoder is responsible for generating the new audio sample. The encoder gradually reduces the resolution of the input audio, extracting features at different levels of abstraction. The decoder then takes these features and upsamples them, gradually increasing the resolution to generate the final audio sample.</p><figure><img alt="Attention U-Net" src="https://cdn-images-1.medium.com/max/1024/1*dJEXcNMlcuG1phcPdX9Vfg.jpeg" /><figcaption>Attention U-Net (Source: <a href="https://arxiv.org/abs/1804.03999v3">Attention U-Net: Learning Where to Look for the Pancreas (</a><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Oktay%2C+O">Oktay, et al.)</a>)</figcaption></figure><p>This U-Net also contains self-attention blocks at the lower levels which help maintain the temporal consistency of the output. It is critical for the audio to be downsampled sufficiently to maintain efficiency for sampling during the diffusion process as well as avoid overloading the attention blocks. The model leverages <a href="https://arxiv.org/abs/2202.00512">V-Diffusion</a> which is a diffusion technique inspired by <a href="https://arxiv.org/abs/2010.02502v4">DDIM</a> sampling.</p><p>To avoid running out of GPU VRAM, the length of the data that the base model was to be trained on needed to be short. Because of this, I decided to train one-shot drum samples due to their inherently short context lengths. After many iterations, the base model length was determined to be 32,768 samples @ 44.1kHz in stereo, which results in approximately 0.75 seconds. This may seem particularly short, but it is plenty of time for most drum samples.</p><h4>Transforms</h4><p>To downsample the audio enough for the attention blocks, several pre-processing transforms were attempted. The hope was that if the audio data could be downsampled without losing significant information prior to training the model, then the number of nodes (neurons) and layers could be maximized without increasing the GPU memory load.</p><p>The first transform attempted was a version of “patching”. Originally proposed for <a href="https://arxiv.org/abs/2207.04316v1">images</a>, this process was adapted to audio for our purposes. The input audio sample is grouped by sequential time steps into chunks that are then transposed into channels. This process could then be reversed at the output of the U-Net to un-chunk the audio back to its full length. The un-chunking process created aliasing issues, however, resulting in undesirable high frequency artifacts in the generated audio.</p><p>The second transform attempted, proposed by <a href="https://arxiv.org/abs/2301.13267">Schneider</a>, is called a “Learned Transform” which consists of single convolutional blocks with large kernel sizes and strides at the start and end of the U-Net. Multiple kernel sizes and strides were attempted (16, 32, 64) coupled with accompanying model variations to appropriately downsample the audio. Again, however, this resulted in aliasing issues in the generated audio, though not as prevalent as the patching transform.</p><p>Because of this, I decided that the model architecture would need to be adjusted to accommodate the raw audio with no pre-processing transforms to produce sufficient quality outputs.</p><p>This required extending the number of layers within the U-Net to avoid downsampling too quickly and losing important features along the way. After multiple iterations, the best architecture resulted in downsampling by only 2 at each layer. While this required a reduction in the number of nodes per layer, it ultimately produced the best results. Detailed information about the exact number of U-Net levels, layers, nodes, attention features, etc. can be found in the <a href="https://github.com/crlandsc/tiny-audio-diffusion/blob/main/exp/drum_diffusion.yaml">configuration file</a> in the <a href="https://github.com/crlandsc/tiny-audio-diffusion">tiny-audio-diffusion</a> repository on GitHub.</p><h3><strong>Conclusion</strong></h3><h4>Pre-Trained Models</h4><p>I trained 4 separate unconditional models to produce kicks, snare drums, hi-hats, and percussion (all drum sounds). The datasets used for training were small free one-shot samples that I had collected for my music production workflows (all open-source). Larger, more varied datasets would improve the quality and diversity of each model’s generated outputs. The models were trained for a various number of steps and epochs depending on the size of each dataset.</p><p>Pre-trained models are available for download on <a href="https://huggingface.co/crlandsc">Hugging Face</a>. See the training progress and output samples logged at <a href="https://wandb.ai/crlandsc/unconditional-drum-diffusion?workspace=user-crlandsc">Weights &amp; Biases</a>.</p><h4>Results</h4><p>Overall, the quality of the output is quite high in spite of the reduced size of the models. However, there is still some slight high frequency “hiss” remaining, which is likely due to the limited size of the model. This can be seen in the small amount of noise remaining in the waveforms below. Most samples generated are crisp, maintaining transients and broadband timbral characteristics. Sometimes the models add extra noise toward the end of the sample, and this is likely a cost of the limit of layers and nodes of the model.</p><p>Listen to some output samples from the models <a href="https://github.com/crlandsc/tiny-audio-diffusion/tree/main/samples">here</a>. Example outputs from each model are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_3MFtFeRAq4yDe8C0BD7vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RUidJCJlxXwx7ohILASagg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxpsup8tNYVbTCHyavJoCg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*djvs4BcSiQEz4H9K3-rLDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y0kGYFQlcsAewrF6zc5lng.png" /></figure><h4>Discussion</h4><p>Along with exploring waveform diffusion models on my local hardware, an important goal for this project was to be able to share that same opportunity with others. I wanted to offer an easy entry point for those with limited resources that were looking to experiment with audio waveform diffusion. Because of this, I structured the project repository to offer step-by-step instructions on how to train or fine-tune your own models as well as generate new samples from the <a href="https://github.com/crlandsc/tiny-audio-diffusion/blob/main/Inference.ipynb">Inference.ipynb</a> notebook.</p><p>In addition, I recorded a <a href="https://youtu.be/m6Eh2srtTro">Tutorial Video</a> that walks through setting up an Anaconda environment and demonstrates ways to generate unique samples with the pre-trained models.</p><p>It is an exciting time for generative audio, especially with diffusion. I have learned an immense amount through building this project and have further expanded my optimism about what is to come in audio AI. I hope that this project can be of use to others looking to explore the world of audio AI as well.</p><p><em>All images, unless otherwise noted, are by the author.</em></p><p><strong>tiny-audio-diffusion code found here: </strong><a href="https://github.com/crlandsc/tiny-audio-diffusion"><strong>https://github.com/crlandsc/tiny- audio-diffusion</strong></a></p><p>Tutorial video on setting up your environment to generate samples with tiny-audio-diffusion: <a href="https://youtu.be/m6Eh2srtTro">https://youtu.be/m6Eh2srtTro</a></p><p>I am an audio scientist with a focus on AI/ML and spatial audio as well as a lifelong musician. If you are interested in more audio AI applications, see my recent article on <a href="https://medium.com/rock-nheavy/the-music-demixing-ai-revolution-9d1528c6ef7c">Music Demixing</a>.</p><p>Find me on <a href="https://www.linkedin.com/in/christopher-landschoot/">LinkedIn</a> &amp; <a href="https://github.com/crlandsc">GitHub</a> and keep up to date with my current work and research here: <a href="https://www.chrislandschoot.com/">www.chrislandschoot.com</a></p><p>Find my music on <a href="https://open.spotify.com/artist/2i6noWJnJQPXPsudoiJuMS?si=fvLOxUPqTAWKB894WXMz5Q">Spotify</a>, <a href="https://music.apple.com/us/artist/after-august/259370281">Apple Music</a>, <a href="https://www.youtube.com/AfterAugust">YouTube</a>, <a href="https://soundcloud.com/after-august">SoundCloud</a>, and other streaming platforms as <a href="https://www.instagram.com/the_after_august/">After August</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddc19e90af9b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/tiny-audio-diffusion-ddc19e90af9b">Tiny Audio Diffusion: Waveform Diffusion That Doesn’t Require Cloud Computing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Music “Demixing” AI Revolution]]></title>
            <link>https://rocknheavy.net/the-music-demixing-ai-revolution-9d1528c6ef7c?source=rss-b64548f914a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d1528c6ef7c</guid>
            <category><![CDATA[audio-source-separation]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[music]]></category>
            <dc:creator><![CDATA[Christopher Landschoot]]></dc:creator>
            <pubDate>Sun, 11 Jun 2023 21:42:56 GMT</pubDate>
            <atom:updated>2023-06-13T20:11:43.688Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Instruments" src="https://cdn-images-1.medium.com/max/1024/1*6rUZog3QNT61gaTItNkHVg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@seemoris?utm_source=medium&amp;utm_medium=referral">Caleb George</a></figcaption></figure><h4><strong>Exploring Advances in Audio Source Separation With a Band-Split Recurrent Neural Network</strong></h4><p>Remixes have been around for nearly 50 years, <a href="https://abbeyroadinstitute.com.au/blog/samples-creativity-copyright-a-history-of-remixing/">originating in the mid-1970s</a> and slowly gaining popularity with various musical movements and technological advancements. Music <a href="https://en.wikipedia.org/wiki/Remix#:~:text=Modern%20remixing%20had%20its%20roots,the%20tastes%20of%20their%20audience.">remixes</a> can broadly be defined as creating a new song or composition from the derivatives of previous creations.</p><p>It wasn’t until the 2010s that the full public acceptance of remixes finally arrived with the popularization of electronic music and new digital tools that made it easier than ever to splice and edit old songs and audio clips in almost any way imaginable. It became common for artists to actually offer their <a href="https://en.wikipedia.org/wiki/Stem_(audio)"><em>stems</em></a> to the public to facilitate the creation of remixes of their songs. Stems are the individual tracks, such as vocals or drums, that make up a fully mixed song. However, there still existed a boundary to remixing tracks when the isolated stems were not accessible. Therefore a need arose to be able to “demix” a track into the individual stems of the song.</p><p>While there are certainly many other factors outside this piece&#39;s scope, this need to isolate stems coupled with the ubiquity of remixes prompted an increase in research and development of music demixing, a.k.a. <em>audio source separation</em>, technologies. These new demixing capabilities also raise an ethical dilemma of deconstructing a song without an artist&#39;s consent, but we will leave that conversation for another day.</p><p>Traditionally, various methods of filtering and trimming were used to cut out or emphasize certain instruments. But, with the evolution of <a href="https://en.wikipedia.org/wiki/Machine_learning">machine learning</a> (ML) over the recent years, specifically the subset utilizing neural networks (a.k.a <a href="https://en.wikipedia.org/wiki/Deep_learning">deep learning</a>), the ability to separate instruments from a fully mixed track has dramatically improved.</p><p>Recently, the buzz of music AI has been largely directed toward music generation, with so many incredible new models being published in the past few years (<a href="https://arxiv.org/abs/2005.00341">Jukebox</a>, <a href="https://arxiv.org/abs/2111.05011">RAVE</a>, <a href="https://www.riffusion.com/">Riffusion</a>, <a href="https://arxiv.org/abs/2301.11325">MusicLM</a>, <a href="https://ai.honu.io/papers/musicgen/">MusicGen</a>, <a href="https://arxiv.org/abs/2208.08706">Musika</a>, <a href="https://arxiv.org/abs/2301.11757">Moûsai</a>, etc). But since 2018, another quiet revolution in the music world has been taking place: <strong>The Music Demixing AI Revolution</strong>.</p><figure><img alt="Audio Source Separation Methods" src="https://cdn-images-1.medium.com/max/1024/0*fAti1BHxKJpMjJEx.png" /><figcaption><em>Audio Source Separation Methods (</em><a href="https://paperswithcode.com/sota/music-source-separation-on-musdb18"><em>source: paperswithcode.com</em></a><em>)</em></figcaption></figure><p>The plot above shows the performance of various state-of-the-art (SOTA) models since 2018. The scoring metric, SDR, stands for Source-to-Distortion Ratio and it effectively measures the quality of the separated audio signal from the <a href="https://en.wikipedia.org/wiki/Ground_truth#:~:text=Ground%20truth%20is%20information%20that,to%20information%20provided%20by%20inference.">ground truth</a>. For demonstrative purposes, the understanding of the absolute metric is less important than using it comparatively. It can be seen that the performance of music demixing models has improved nearly 3-fold in less than 5 years. Before then, the task largely wasn’t feasible. This is simply remarkable.</p><p>Much of this research is still experimental and has not made its way into producer tools. However, some algorithms, such as <a href="https://github.com/deezer/spleeter">Deezer Spleeter</a>, have been incorporated into powerful tools utilized by producers all over the world:</p><ul><li><a href="https://www.izotope.com/en/products/rx.html">iZotope’s</a> music rebalance feature in RX</li><li><a href="https://www.steinberg.net/spectralayers/">Steinberg’s</a> unmix feature in SpectralLayers</li><li><a href="https://acondigital.com/products/acoustica">Acon Digital’s</a> remix tool in Acoustica</li><li><a href="https://www.virtualdj.com/stems/">VirtualDJ’s</a> stem isolation feature</li><li><a href="https://www.algoriddim.com/">Algoriddim’s</a> music source separation feature in NeuralMix and djayPRO</li></ul><p>But even since the incorporation of Spleeter (avg. SDR: 5.91) into these products several years ago, research has improved audio source separation performance by over 60%.</p><p>As a musician that also lives in the world of audio AI tech, these rapid improvements have been incredibly exciting. So I decided to join in on the development by participating in <a href="https://www.aicrowd.com/challenges/sound-demixing-challenge-2023/problems/music-demixing-track-mdx-23">AIcrowd’s Music Demixing Challenge 2023</a>. This challenge, supported by <a href="https://www.sony.com/en/">Sony</a>, <a href="https://moises.ai/">Moises</a>, and <a href="https://www.mitsubishielectric.com/en/index.html">Mitsubishi Electric</a>, was created to promote research and development in audio source separation technologies. Through my participation, I learned a great deal about these cutting-edge music demixing algorithms as well as the shortcomings that the technology still has.</p><p>I chose to build a <a href="https://github.com/crlandsc/Music-Demixing-with-Band-Split-RNN">Band-Split Recurrent Neural Network (BSRNN)</a> for the competition based on a <a href="https://arxiv.org/abs/2209.15174">paper</a> published in 2022 by Luo &amp; Yu. I was initially drawn to this framework over others because according to the paper, it outperforms all other models when using the same training dataset. This is likely because contrary to other models, it offers a way to bias the model with human knowledge about the characteristics of each instrument. We can design it to put more importance on frequency bands that are more relevant to one instrument vs another.</p><h3>Background (Digital Audio Basics)</h3><p>Before getting into the nitty-gritty of this complex deep learning framework, we need to discuss how audio data is processed in order for computers to be able to work with it.</p><p>Physically, sound is made up of pressure oscillations of air particles over time. To translate this to a computer’s language, we can capture these oscillations with microphones and map them to a digital signal that oscillates between -1 and 1. This can be seen as the “raw” audio waveform and it is generally how computers store uncompressed audio data. This is <a href="https://en.wikipedia.org/wiki/Time_domain#:~:text=Time%20domain%20refers%20to%20the,data%2C%20with%20respect%20to%20time."><em>time-domain</em></a> data as it measures amplitude over time. We can see an example of this below in a 7-second sample of a vocal recording.</p><figure><img alt="Waveform plot" src="https://cdn-images-1.medium.com/max/647/1*zPgmlCifOQ_3sB_C7J6RTg.png" /><figcaption>Waveform (Time Domain)</figcaption></figure><p>While this time-domain representation is analogous to how we actually hear sound, it is hard to glean any information from visually inspecting it. Sound can also be viewed in another manner in the <a href="https://en.wikipedia.org/wiki/Frequency_domain"><em>frequency-domain</em></a>. To do this, the waveform is transformed into the magnitude of its frequency components by using the <a href="https://www.youtube.com/watch?v=spUNpyF58BY">Fourier Transform</a>. This basically shows the measure of how loud the sound is at each frequency, or in musical terms at each pitch. We can see the same vocal clip from above transformed into the frequency-domain below. Here we can gather a little more information from simple visual inspection, such as a majority of sound energy for this vocal track exists between 250Hz and 2kHz.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*91hNx-beYQvGpgTTVVxTZQ.png" /><figcaption>Spectrum (Frequency Domain)</figcaption></figure><p>This information is very useful in many contexts. However, we have now lost all temporal information from the data during this transformation.</p><p>Lucky for us, there is a way to combine information from both the frequency-domain and time-domain. This comes in the form of a <a href="https://en.wikipedia.org/wiki/Spectrogram"><em>spectrogram</em></a>. A spectrogram acts like a heatmap of the sound over time, with time along the x-axis and frequency along the y-axis. The same vocal track we have been working with is plotted below in the form of a spectrogram. We can now see clear melody patterns by looking at the yellow stripes moving up and down as they run horizontally across the spectrogram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*KjTLwEbm1YWw0_gFXnJcrQ.png" /><figcaption>Spectrogram (Time-Frequency Domain)</figcaption></figure><p>Computers can work with data in all of these formats, but often times the spectrogram is the preferred method as it offers a good balance between compressing the data without losing too much information. Additionally, a spectrogram representation is similar to image data, which computers are very good at working with. Nothing is free, however, so this compression and translation from audio to spectrogram introduce some drawbacks, such as loss of resolution and phase information. But for our music demixing purposes, the benefits of spectrogram data outweigh the costs.</p><h3>Model Architecture</h3><p>Now that we have discussed the basics of how sound is transformed into a format that our model can work with, let’s get into the structure and how it functions.</p><p>The BSRNN model is split into 3 separate modules, each with a distinct function.</p><ol><li>Band-split module</li><li>Band and Sequence Separation Modeling Module</li><li>Mask Estimation Module</li></ol><p>Because the BSRNN is a complex framework, we will step through each module individually in detail to describe functionality in a digestible manner.</p><h4>1. Band-Split Module</h4><figure><img alt="Band-Split Module" src="https://cdn-images-1.medium.com/max/1024/1*n3H5oioHzRFzGJdGImFXKw.png" /><figcaption>Band-Split Module</figcaption></figure><p>In the first module, small chunks of the raw audio signal are converted to a spectrogram via the <a href="https://en.wikipedia.org/wiki/Short-time_Fourier_transform">short-time Fourier transform</a> (STFT) and fed through the network one at a time. Each spectrogram is then split into <em>sub-bands</em> at the “Band Split” step in the diagram. These sub-bands can be thought of as little horizontal slices of the spectrogram image. This allows the network to adjust the weights according to each “slice” of the spectrogram. It would follow that a higher number of sub-bands would mean an increase in resolution as well as an increase in the size of the model — a hard tradeoff to balance.</p><p>This is a key piece of the BSRNN framework, as the number of sub-bands can be adjusted to bias the system depending on what the target instrument is.</p><p>For example, most sound energy for a bass guitar exists in frequencies below 200Hz, so we can bias our model to include more sub-bands below 200Hz and fewer sub-bands above 200Hz. This effectively allows the model to “focus” where we know the important frequencies for that instrument reside, providing an elegant balance of resolution and model size.</p><p>The number of sub-bands and their bandwidth can be fine-tuned for each target instrument. Sub-bands are then normalized and sent through one fully-connected dense layer per band. The dense layers are then merged to be sent through the next module.</p><h4>2. Band and Sequence Separation Modeling Module</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zcmXJxCUETW7mWdG1lGoCg.png" /><figcaption>Band and Sequence Separation Modeling Module</figcaption></figure><p>In the second module, the data is fed through a pair of <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network">recurrent neural networks</a> (RNNs), each serving a different purpose. Both RNNs consist of a normalization step, a <a href="https://analyticsindiamag.com/complete-guide-to-bidirectional-lstm-with-python-codes/">bi-directional LSTM</a> (long short term memory network), and a fully connected dense layer.</p><p>The first RNN is fed with the data across the time dimension. To conceptualize this process, this can be thought of as the RNN stepping through the spectrogram image from left to right (across the time axis). This allows the network to learn important temporal aspects of the sound. The data is then summed with the pre-RNN data before entering the second RNN.</p><p>The second RNN processes the data across frequency bands. Supplementary to the first RNN, this can be conceptualized as the RNN stepping through the spectrogram from bottom to top (across the frequency axis). This allows the network to learn important spectral aspects of the sound. Similar to the first RNN, the pre and post-RNN data are then combined.</p><p>This pair of RNNs can be repeated multiple times to improve the complexities that the network can learn. For example, my implementation used 12 layers to align with the <a href="https://arxiv.org/abs/2209.15174">BSRNN paper</a> specifications.</p><p>Structuring the network to be able to learn from the data across both of these axes creates the opportunity for a more holistic understanding of the audio. After all of the RNN layers, the data then enters the final module.</p><h4>3. Mask Estimation Module</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omrK4yaulchnNMcICZkCMg.png" /><figcaption>Mask Estimation Module</figcaption></figure><p>The final module follows a similar pattern to the first module. The data is band-split and normalized, but instead of a single fully connected layer, it is then fed through a multi-layer perceptron (MLP) network. An MLP is essentially a mini-feed-forward network with an input layer, hidden layers (1 layer in our case), and an output layer. This MLP implementation utilized the hyperbolic tangent activation function. For more information about MLPs, see my previous blog post <a href="https://medium.com/@crlandschoot/how-do-neural-networks-really-work-ac137e0b052">here</a>.</p><p>Again, the sub-bands are then merged, resulting in a spectrogram “mask”. This mask is then multiplied by the original signal to generate the target spectrogram of “unmixed” audio. This target is then compared to the ground truth and the loss is calculated and backpropagated through the network to update all of the weights.</p><h4>Full BSRNN Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vS0he1gIxPQhtU88IS1_cQ.png" /><figcaption>Full Band-Split Recurrent Neural Network Architecture</figcaption></figure><h3>Conclusions</h3><p>This Band-Split RNN framework offers state-of-the-art performance while maintaining a modest model size. This is because the model can be biased with previous knowledge about the characteristics of instruments to balance the tradeoff of resolution vs model size. As a recently published novel framework, there is still a great deal of possible model tuning, particularly regarding instruments other than vocals.</p><h3>Limitations &amp; Future Work</h3><p>While the performance of the BSRNN and other SOTA models are impressive, there are still many limitations for this technology.</p><p>The biggest current challenge is that the types of instruments separated are typically limited to vocals, bass, drums, and “other” (grouping of all other instruments). Some limited models have been trained for piano and other specific instruments, but they are few and far between. Even these additional instruments do not come close to capturing the wide array of instruments that exist. This is largely due to the fact that there is limited training data for each different type of instrument as well as the fact that a separate model would need to be trained for each instrument.</p><p>There are two paths to overcoming these challenges. The first would be to curate and publish open-source datasets of mixed music and stems that also include other types of instruments. If sufficient data exists, there is no reason that the BSRNN (or other SOTA models) couldn’t be trained on it. This has not been largely pursued yet, as it is a significant task that will require a great deal of time and resources.</p><p>The second would be to research new frameworks that allow for more flexibility in how they are trained and the sources they can separate. For example, it would be conceivable to develop a method of training a model on ONLY the stems of a wide range of instruments which would not require fully mixed songs. This would dramatically increase the availability of audio samples to train a myriad of other instruments. Recent work by <a href="https://arxiv.org/abs/2204.09079">Zhu et al.</a> proposes a method for this.</p><p>Another area of recent interest is using generative models for source separation. <a href="https://arxiv.org/abs/2302.02257">Mariani et al.</a> propose a <em>Multi-Source Diffusion Model</em> (MSDM) that leverages a single diffusion model to separate multiple different sources. This attacks the problem of having to train separate models for each source, allowing for a much more flexible design.</p><p>The past 5 years have seen an incredible advancement in audio source separation technology, though there are still significant challenges for this technology to overcome. With the increased interest in all things AI currently and the influx of funding for R&amp;D, this trend is likely to continue.</p><p>It is an exciting time to be in the audio and music AI space as each day brings new developments that were mere figments of our imaginations not long ago.</p><p><strong>BSRNN MDX-23 Code found here: </strong><a href="https://github.com/crlandsc/Music-Demixing-with-Band-Split-RNN"><strong>https://github.com/crlandsc/Music-Demixing-with-Band-Split-RNN</strong></a></p><p>A huge thank you to <a href="https://www.linkedin.com/in/amanteur/">Amantur Amatov</a> for the initial unofficial public implementation of the BSRNN framework!</p><p>I am an audio scientist with a focus on AI/ML and spatial audio as well as a lifelong musician.</p><p>Find me on <a href="https://www.linkedin.com/in/christopher-landschoot/">LinkedIn</a> and keep up to date with my current work and research here: <a href="https://www.chrislandschoot.com/">https://www.chrislandschoot.com/</a></p><p>Find my music on <a href="https://open.spotify.com/artist/2i6noWJnJQPXPsudoiJuMS?si=fvLOxUPqTAWKB894WXMz5Q">Spotify</a>, <a href="https://music.apple.com/us/artist/after-august/259370281">Apple Music</a>, <a href="https://www.youtube.com/AfterAugust">YouTube</a>, <a href="https://soundcloud.com/after-august">SoundCloud</a>, and other streaming platforms as <a href="https://www.instagram.com/the_after_august/">After August</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d1528c6ef7c" width="1" height="1" alt=""><hr><p><a href="https://rocknheavy.net/the-music-demixing-ai-revolution-9d1528c6ef7c">The Music “Demixing” AI Revolution</a> was originally published in <a href="https://rocknheavy.net/">Rock n’ Heavy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How do Neural Networks REALLY Work?]]></title>
            <link>https://medium.com/@crlandschoot/how-do-neural-networks-really-work-ac137e0b052?source=rss-b64548f914a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac137e0b052</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Christopher Landschoot]]></dc:creator>
            <pubDate>Sat, 18 Mar 2023 21:23:49 GMT</pubDate>
            <atom:updated>2023-03-19T21:07:48.864Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="https://unsplash.com/photos/ZiQkhI7417A" src="https://cdn-images-1.medium.com/max/1024/1*C_6Nzuy2ZJrurWOUxks4Tw.jpeg" /><figcaption>Image Source: <a href="https://unsplash.com/photos/ZiQkhI7417A">Alina Grubnyak</a></figcaption></figure><p>The world seems to have fallen in love with neural networks over recent years. Who can blame them? It’s an incredible technological innovation that <a href="https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414#:~:text=Neural%20networks%20were%20first%20proposed,the%20first%20cognitive%20science%20department.">lived in theory books</a> for years before computing power caught up to make it possible.</p><p>“Machine learning” and “neural networks” have been picked up as buzzwords by many corporations to the point where even some tech-averse people have heard of them. And while these words keep afloat massive marketing campaigns, and make some panic about the <a href="https://en.wikipedia.org/wiki/Technological_singularity">Singularity</a>, the algorithms themselves are more than just tech bro jargon. What many see as this magic black box that spits out answers to problems actually has its roots in fairly straightforward logic and math.</p><p>Of course, most techies would tell you a neural net is supposed to simulate the processes of the human brain with some neurons organized into layers, passing information to one another. While this analogy may be somewhat true at a high level, anything more than this generalization would be <a href="https://medium.com/swlh/do-neural-networks-really-work-like-neurons-667859dbfb4f">considered a stretch</a>. In addition to this misconception, many people that utilize this incredible technology don’t actually know what’s happening under the hood. While I’m not going to build another <a href="https://openai.com/blog/chatgpt">ChatGPT</a> in this piece (your secret is safe with me <a href="https://en.wikipedia.org/wiki/Sam_Altman">Sam Altman</a>), I hope to help illuminate some of the basics of how neural networks actually work.</p><figure><img alt="ChatGPT" src="https://cdn-images-1.medium.com/max/564/1*auF_slDRMg2ioOf2gI68tA.gif" /></figure><p>I’d like to share a basic example of a Multi-Layer Perceptron (MLP) Neural Network that I found when watching the wonderful Valerio Velardo’s <a href="https://www.youtube.com/c/ValerioVelardoTheSoundofAI">Sound of AI YouTube channel</a>. The name may sound scary, but it’s really not that bad once it’s broken down. I chose this example because MLPs can be a very simple type of “Dense” (aka <em>feedforward</em>) neural network, which basically means that data only flows through the network in one direction: from input to output.</p><p>Supporting code that implements the concepts of this piece can be found <a href="https://github.com/crlandsc/Multi-Layer-Perceptron-Neural-Network">here</a>. The code is split into two examples:</p><ol><li>How to build a simple MLP neural network from scratch.</li><li>How to build a simple MLP neural network with <a href="https://www.tensorflow.org/api_docs/python/tf/keras">TensorFlow (Keras)</a>.</li></ol><p><strong>Neural Networks (Generally…)</strong></p><p>Let’s begin by conceptualizing a few key things. Neural networks are indeed made up of a bunch of interconnected neurons (aka <em>nodes</em>). At its base level, a node can be defined as something that receives data, performs a computation on the data, and then sends along the processed data. You can think of it like a cashier receiving a payment, subtracting the price from the cash, and returning the change to the customer.</p><figure><img alt="Cash Register" src="https://cdn-images-1.medium.com/max/480/1*kwzDGU4i_FQZIolMTtumcg.gif" /></figure><p>This action from a single cashier doesn’t seem too special (sorry cashiers). But if you chain together an entire city of cashiers, you have a massive flow of commerce that one person could never fully comprehend. The real power lies in the number of these cashiers and how money flows through the city between them. The same goes for nodes.</p><p>These nodes (our cashiers) are organized into various layers which define how they are connected with one another. The image below provides a good visualization of this process. In our example, data will follow the flow of the arrows in the diagram, traveling from left to right, passing through each layer. Each node will receive the data, perform a calculation, and then send it along.</p><figure><img alt="Neural Network" src="https://cdn-images-1.medium.com/max/441/1*W34pcjmh_9RNMveGIrrXpw.jpeg" /><figcaption>Image Source: <a href="https://www.geeksforgeeks.org/multi-layer-perceptron-learning-in-tensorflow/">https://www.geeksforgeeks.org/multi-layer-perceptron-learning-in-tensorflow/</a></figcaption></figure><p>There is one final piece to the puzzle related to the connections between each node. Not all connections between nodes are created equal, some are stronger than others. This diversity in the connection strength between nodes is the secret sauce that allows a neural network to learn complex trends from data.</p><p>Now that we have some basic concepts, let’s dive into the technical details.</p><p><strong>Multi-Layer Perceptron (MLP) Neural Network</strong></p><p>An MLP consists of three types of layers: the input layer, the hidden layer(s), and the output layer. The input layer receives the raw data, while the output layer produces the final prediction. An arbitrary number of hidden layers process the information and are responsible for the bulk of the network’s computation. Adjusting the number of hidden layers and nodes at each layer is an important means of tuning a neural network.</p><p>Each node in the MLP is connected to nodes in the previous layer and the next layer. The strength of these connections is represented by <em>weights</em>, which are automatically adjusted as the model trains and learns to optimize the network’s performance.</p><p>Let’s step through how data flows through the network, beginning with <em>forward propagation</em>.</p><p><strong>Forward Propagation</strong></p><p>The process of <em>forward propagation</em> involves passing the input data through the network and calculating the output. As data is transferred from one node to the next, it is multiplied by the <em>weight</em> between the nodes. Once each node in the hidden layer(s) receives the data input from the previous layer, it applies an <em>activation function</em> to produce an output. This output is then passed on to the next layer(s) until it reaches the output layer, where final predictions are made.</p><p><strong>Backpropagation</strong></p><p>So now that we have made a prediction, we’ve made a neural network and we’re done, right? Not quite. The purpose of a neural network, and machine learning more generally, is for the model to update itself (or “learn”) to get better at predicting the correct answer.</p><p>Because we have predictions, we can compare them to the data that we are using to train the model to determine how far off we were from the actual answer, also known as the <em>ground truth</em>. Then we need to use this information to adjust the <em>weights</em> between each node through a process known as <em>backpropagation</em>.</p><p>In technical terms, this is done using the chain rule of calculus, taking the derivative of the <em>activation function</em>, which allows us to calculate the gradient of the error with respect to each <em>weight</em>.</p><p><strong>Gradient Descent with Mean Squared Error (MSE)</strong></p><p>In order to adjust the <em>weights</em> to minimize the error, we use an optimization algorithm called <em>gradient descent</em>. <em>Gradient descent</em> works by iteratively adjusting the <em>weights</em> in the direction of the steepest descent of the error function.</p><figure><img alt="Gadient Descent" src="https://cdn-images-1.medium.com/max/500/1*nlvblFZKBOJVCyb7yDuQlQ.png" /><figcaption>Image Source: <a href="https://www.javatpoint.com/gradient-descent-in-machine-learning">https://www.javatpoint.com/gradient-descent-in-machine-learning</a></figcaption></figure><p>Breaking down the terminology, <em>gradient</em>=slope and <em>descent</em>=downward. So in simpler terms, <em>gradient descent </em>could be translated to mean “downward slope”.</p><p>Basically, you can think of it like a ball rolling down a hill toward the bottom of a valley. The ball is our current set of <em>weights</em> and each point on the hill represents a different set of <em>weights</em> for all node connections. At the bottom of the valley is our lowest error, aka our best-performing model. Our goal is for this ball to reach the bottom of the valley, so at each step (or iteration) we look to see which way the ball would be rolling down. Then we adjust all of the <em>weights</em> accordingly in that direction until we achieve our optimal model.</p><p>Many different error metrics can be used with <em>gradient descent</em> depending on the problem at hand. For example, the <em>mean squared error (MSE)</em> is a commonly used metric for regression (i.e. number prediction) problems.</p><p><strong>Training the Model</strong></p><p>Now that we have our process set up, we can begin training our model. We start by initializing the <em>weights</em> randomly. We then pass the input data through the network using <em>forward propagation</em> and calculate the error of the output using a metric such as <em>MSE</em>. We then use <em>backpropagation</em> to calculate the gradients of the error with respect to each <em>weight</em> and adjust the <em>weights</em> using <em>gradient descent</em>.</p><p>This process is repeated for a fixed number of iterations, or until the error is minimized to a satisfactory level. The number of nodes within each hidden layer and the number of hidden layers can be tuned to achieve optimal performance for a specific problem. However, increasing the number of hidden layers or the size of the hidden layers can lead to <em>overfitting</em>, which essentially means that the model is memorizing the training data rather than learning the underlying patterns in the data.</p><p>To prevent <em>overfitting</em>, techniques such as <em>regularization</em>, <em>early stopping</em>, and <em>dropout</em> can be applied. <em>Regularization</em> (less common than other methods) adds a penalty term to the error function to discourage large <em>weights</em>. <em>Early stopping</em> halts the training process when the error on the validation dataset starts to increase. <em>Dropout</em> randomly drops out nodes during training to prevent them from relying too heavily on each other.</p><p>Once the model is trained, it can be used to make predictions on new, unseen data. The new input data is passed through the network using <em>forward propagation</em>, and the output is calculated. The prediction is then compared to the true output to evaluate the performance of the model.</p><p><strong>Recap</strong></p><p>In summary, neural networks are powerful machine learning algorithms that can be used for a wide range of tasks, such as image recognition, natural language processing, and time series forecasting. Various other forms of neural networks, such as <em>convolutional neural networks (CNNs)</em> or <em>recurrent neural networks (RNNs)</em>, can be leveraged for better performance regarding certain tasks.</p><p>The processes of <em>forward propagation</em> and <em>backpropagation</em> while utilizing error <em>gradient descent</em>, allow neural networks to learn from data and adjust their <em>weights</em> to optimize performance. While many aspects of neural networks can vary and become far more complicated, all neural networks are built upon this set of basic processes.</p><p>Understanding the underlying concepts of any powerful tool, rather than just memorizing syntax, can allow us to maximally leverage and optimize these technologies to solve our problems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac137e0b052" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Does Anyone Remember Parks and Rec?]]></title>
            <link>https://medium.com/@crlandschoot/does-anyone-remember-parks-and-rec-19d13f917d04?source=rss-b64548f914a5------2</link>
            <guid isPermaLink="false">https://medium.com/p/19d13f917d04</guid>
            <category><![CDATA[parks-and-recreation]]></category>
            <category><![CDATA[america]]></category>
            <category><![CDATA[free-speech]]></category>
            <dc:creator><![CDATA[Christopher Landschoot]]></dc:creator>
            <pubDate>Mon, 03 May 2021 05:02:15 GMT</pubDate>
            <atom:updated>2021-05-03T05:02:15.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LVodAhephiM1gqRTHWjaKA.png" /></figure><p>If you’re like me, of course you do. That’s a silly question. It was one of the most successful and beloved shows (especially among the younger crowd, like me) on TV from 2009 to 2015. It is still one of my favorite shows and I have re-watched all 7 seasons from start to finish multiple times. There were myriad reasons it captured the hearts of so many viewers: The cast was phenomenal (Amy Poehler, Nick Offerman, Chris Pratt, Aziz Ansari, etc.), the writing was spectacular (Michael Schur — <em>The Office, Brooklyn 99, The Good Place</em>), the characters were relatable on many levels, it made the mockumentary funnier and less awkward (don’t hate me diehard <em>The Office</em> fans), etc. So why am I focusing on a show that has been off the air for 6 years?</p><p>It popped into my head the other day, seemingly out of nowhere, and a feeling of nostalgia and longing swept over me. Because this ostensibly never-ending pandemic has left me with a great deal of time alone with my thoughts, I decided to take the day to investigate why I was feeling this way. It could have been that I missed Leslie’s (Amy Poehler) undying positive attitude in this time of nearly ubiquitously negative media, Ron’s (Nick Offerman) hilariously dry and direct approach to life (and eating meat), April (Aubrey Plaza) and Andy’s (Chris Pratt) uniquely perfect relationship, Ben’s (Adam Scott) nerdiness, Tom’s (Aziz Ansari) swagger, or just the general sense of levity that the show could bring to even the rainiest of days. While each of those thoughts brought a brief smirk to my face, it didn’t feel like the real reason I had such a sense of longing for the show.</p><p>It was only when I thought about the relationship between Leslie and Ron that underpinned the entire series that I knew I had found the true source of my yearning to pop down on the couch and turn on an episode. They were a couple of the funniest and most likable characters I have ever watched on TV as Poehler and Offerman’s chemistry was magic.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FR55sOeBR22ogg%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FR55sOeBR22ogg%2Fsource.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FR55sOeBR22ogg%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="243" frameborder="0" scrolling="no"><a href="https://medium.com/media/b183883dd02789abe26a881b4d63360c/href">https://medium.com/media/b183883dd02789abe26a881b4d63360c/href</a></iframe><p>Ok, so we established that I really liked two big-time actors on a TV show, that’s not really a revelation, right? Well, that part isn’t, but it is the personalities and development of these characters and how they interacted that made their friendship so compelling.</p><p>Leslie and Ron fall on opposite ends of the spectrum when it comes to many personality traits and political ideologies. Leslie is incredibly energetic, idealistic, and filled with emotion while Ron is more even-keeled, pragmatic, and stoic. Leslie believes in the benefits that government can offer to the people and Ron is ever-skeptical of governmental tyranny. Leslie is clearly on the Left and Ron on the Right. But for what they lacked in common personality traits and political ideology, they made up for in respect, value, and friendship with one another. So much of the show was based on the various ways these two drastically different characters comically butted heads. The two would be placed in some silly conundrum, both would put forth their opinions on how to address the matter, and the resolution would typically result from a combination of both views and an acknowledgment by both characters of the value of the opposing side’s argument. A clear example that immediately came to mind was Season 5, Episode 16, “Bailout”.</p><p>The basis of the episode is that the local fringe/foreign film shop is going out of business and Leslie, as a councilwoman, wants to use the powers of government to help rescue it, as she views it as a place that offers genuine value to the community. She meets with the film buff store owner and proposes that the store applies for historical landmark status to procure government funding. Overhearing this plan, Ron comments that this intervention is a “thinly veiled government bailout” and he opposes it. While the government funding will help save the business, Leslie instructs the owner that he needs to use the subsidized money to sell “what the people want” in order to stay afloat. This ultimately results in the owner misconstruing Leslie’s directives, turning the store into an adult video store because it was the part of his business that always performed the best. The hilarity of the situation is that although the business had been saved, this means that the <a href="https://youtu.be/esCNsIrpXqc">government now effectively owns a porn shop</a>. A closing scene consists of a conversation between Ron and Leslie discussing the issues of government intervention, but also expressing that there needs to be a way for the government to support places that “provide community value but don’t necessarily rake in the money”. This results in the Pawnee government hosting a weekly movie night at the community center, demonstrating that the government can use its power to provide a positive service to the community. And of course, we can’t forget the part where <a href="https://youtu.be/i7Dgy4p7lsA">Ron orders a party platter of meat</a> for 12 people to eat in 4 minutes. Classic.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FQgXU83mKEuwnu%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FQgXU83mKEuwnu%2Fsource.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FQgXU83mKEuwnu%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/9566ae5d8fd89669f5ad79db4f6f0a4a/href">https://medium.com/media/9566ae5d8fd89669f5ad79db4f6f0a4a/href</a></iframe><p>This is a perfect example of a clash between the ideologies of these characters that doesn’t result in the devolution of their mutual respect or friendship. Furthermore, the solution to the dilemma, itself, demonstrates the value of both points of view. This gets at the core of why these characters are so compelling to me, particularly at this moment in time. This relationship time and time again highlights the value of true diversity, not of immutable characteristics, but of thought. It champions the idea that even though they may not agree on many issues, they are both Americans that value and respect one another. Additionally, they are both open-minded enough to accept when they’re wrong and learn from their mistakes to become better people. This starkly contrasts with the America of 2021, where I don’t know of any relationship between two people of opposing views that is foundationally based on respect between one another. In many cases, it’s hard for me to even name a single person that will have a conversation with someone of an opposing view without dismissing them as immoral or unintelligent.</p><p>If you feel I’m painting with a broad brush, ask yourself if you can openly reveal your ideologies to your friends without losing them? Conversely, if you feel you do openly express your ideologies, are you close with anyone that openly differs from you? Can either of you talk about it without fighting? If you can say yes to any of these questions, that is wonderful and we need more people like you! But, if you’re being honest, my guess is probably not. Unfortunately, this is not the norm in our current culture. However, I do believe there are many people out there on both sides that share these same thoughts, they just don’t know what to do.</p><p>So how are we supposed to fix this?</p><p><strong>The answer lies in restoring mutual respect for one another in order to cultivate a culture of free speech.</strong> To be an America for all people we need to hold the common value that there should be room for all views at the table of public discussion. This constitutional idea seems to have become an unpopular opinion (at least publicly), but it cannot be denied that in the last election 74 million people voted for Trump and 81 million people voted for Biden. I am not commenting on some fringe phenomenon, this affects the vast majority of people in America. There needs to be room in our culture for both Biden voters and Trump voters. This means that we cannot dismiss half the country as racists because they support one side or radicals because they support the other. We need to create a culture that can simultaneously provide respect for both Left and Right views.</p><p>Pursuing this principle is going to be hard and uncomfortable, as it has always been. It is far easier to dismiss the other side as immorally evil than trying to get to know and understand their perspective. This means encouraging discourse of differing ideas, not castigating or silencing them. It means to sit back and genuinely listen instead of building a straw man of your opponent to attack. The only way for society to distinguish the good ideas from the bad ones is to present all ideas, no matter how radical, and iteratively engage in public and private discourse to determine which ones our society wants to live by.</p><p>Throughout my life I have never been a very political or confrontational person, but this pandemic coupled with the ever more polarized state of our nation has forced me to internally confront these topics on almost a daily basis. Needless to say, I have thought about this long and hard before I found the hill that I will die on. As I have watched people across this country move further and further apart, I have realized this is not a path to a better country or sustainable future. I believe the road forward runs through the principles of Voltaire manifested in a culture of free speech. So if I am canceled for my belief in broadening the Overton Window to let in more opinions, then so be it. At least everyone will know where I stand. And I hope that others that feel similarly can find their voice too, as I believe this is the only way to keep our country together.</p><p>I want to have an Amy to my Ron or vice versa. Having to acknowledge other views opens your mind, sharpens you to potential fallacies in your ways of thinking, and improves your understanding of the world. I still have a great deal to learn to inform my views and I am discovering new things that shape my ideas every day. I work hard to continually acknowledge that my way may not be the be-all-end-all, no matter how confident I feel. This is a process that should never end, and it is our duty in contributing to the progress of humanity.</p><p>So if you’re up to it, let’s hit up JJ’s Diner and order a 12-person party platter, and we can have a conversation as friends where you share a little of your views and I share a little of mine and we don’t burn down the entire place. I know that I’d learn something, and I think you might too.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FxUPGclJ5wKOJmENxcs%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FxUPGclJ5wKOJmENxcs%2Fsource.mp4&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FxUPGclJ5wKOJmENxcs%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a95614167b10a22420268427b4268bd/href">https://medium.com/media/5a95614167b10a22420268427b4268bd/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19d13f917d04" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>