<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Peech on Medium]]></title>
        <description><![CDATA[Stories by Peech on Medium]]></description>
        <link>https://medium.com/@peechapp?source=rss-a422e61a20f9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IjXHWuzz70bq4rkxdx0qfQ.png</url>
            <title>Stories by Peech on Medium</title>
            <link>https://medium.com/@peechapp?source=rss-a422e61a20f9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:35:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@peechapp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Future of Text-to-Speech: Market Trends and Opportunities]]></title>
            <link>https://medium.com/@peechapp/the-future-of-text-to-speech-market-trends-and-opportunities-89d1aa6ae33a?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/89d1aa6ae33a</guid>
            <category><![CDATA[market-research]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[tts]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Mon, 02 Dec 2024 23:46:43 GMT</pubDate>
            <atom:updated>2024-12-02T23:52:25.640Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XoCFkBZoEQdJAi3YcbAEGQ.png" /></figure><p>As a startup in the text-to-speech (TTS) industry, we’ve witnessed firsthand the rapid evolution and growing impact of TTS technology. What was once a niche tool for accessibility has now become integral to a multitude of applications — from education and healthcare to entertainment and productivity. In this article, we’ll share insights into the current state of the TTS market, emerging trends, and the vast opportunities that lie ahead, drawing upon our experiences and data at <a href="https://www.getpeech.com/">Peech</a>.</p><h3>Market Overview</h3><h4><strong>Global Text-to-Speech Market Growth</strong></h4><p>The global TTS market is witnessing robust growth, projected to expand from <strong>$3.42 billion in 2024</strong> to approximately <strong>$7.17 billion by 2029</strong>. This represents a <strong>compound annual growth rate (CAGR) of 15.96%</strong>, indicating a strong upward trajectory fueled by technological innovations and a growing emphasis on accessibility across sectors such as education, healthcare, entertainment, and customer service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/0*6WQcGy8ZPPLZ1xbp.jpg" /><figcaption>Global Text-to-Speech Market Size (2024–2032)</figcaption></figure><h4><strong>Global Audiobooks Market</strong></h4><p>Parallel to the TTS market, the global audiobook market has seen remarkable expansion, with revenues reaching approximately <strong>$6.83 billion in 2023</strong>. Projections indicate a CAGR of <strong>26.2%</strong> from 2024 to 2030, aiming for a market size of <strong>$35.04 billion</strong> by 2030. The synergy between TTS advancements and audiobooks popularity is undeniable — this is something that we’ll see getting tighten up in upcoming years.</p><h4><strong>Regional Insights</strong></h4><p><strong>North America:</strong> Holds a significant share of the audiobook market, attributed to the widespread adoption of smartphones and a growing preference for digital content consumption. The U.S. alone accounts for over <strong>60%</strong> of domestic audiobook purchases.</p><p><strong>Europe:</strong> The UK audiobook market represented over <strong>23%</strong> of Europe’s revenue in 2023, with Germany expected to grow at a CAGR of <strong>25%</strong> from 2024 to 2030.</p><h4><strong>Genre Preferences</strong></h4><p><strong>Fiction Audiobooks:</strong> Dominated the market in 2024, accounting for more than <strong>64%</strong> of revenue.</p><p><strong>Non-Fiction:</strong> Anticipated to register the fastest growth, with a CAGR exceeding <strong>27%</strong> from 2024 to 2030.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/0*aZNSv3_AnMUr4aIU.png" /><figcaption>Audiobooks Market Size by Genre</figcaption></figure><p>Understanding the B2C market separately from B2B is crucial because consumer preferences, usage patterns, and monetization strategies differ significantly. While B2B focuses on scalability, integration, and enterprise-level support, B2C centers around user experience, personalization, and affordability. Disclaimer: in both cases — state of the art tech is needed :)</p><h3><strong>B2C Market — For Individuals</strong></h3><h4><strong>Surge in Mobile Applications</strong></h4><p>In the B2C segment, particularly within mobile applications, the TTS market has seen exponential growth. According to Sensor Tower data:</p><p>The <strong>top 10 iOS TTS apps</strong> generated approximately <strong>$240,000 per month</strong> in June 2022. By October 2024, this figure soared to an estimated <strong>$1.35 million per month</strong>. This marks a <strong>5.6x growth</strong> in just over two years, with a staggering <strong>CAGR of approximately 137%</strong>.</p><h4><strong>Competitive Landscape</strong></h4><p>At Peech, we’ve closely monitored the competitive landscape to understand market dynamics and user preferences.</p><p>Companies like <a href="https://elevenlabs.io/">ElevenLabs</a> and <a href="https://play.ai/">PlayAI</a> are leading the way with proprietary synthesis models that deliver superior voice quality. Their focus on innovation allows them to offer more natural and expressive voices, which is increasingly important for user engagement.</p><p><a href="https://speechify.com/?source=fb-for-mobile&amp;via=binh-nguyen-thanh&amp;gad_source=1"><strong>Speechify</strong></a></p><ul><li><strong>Market Position:</strong> Leading player in the B2C TTS market.</li><li><strong>Users:</strong> Serves over <strong>20 million users</strong> across platforms.</li><li><strong>Monthly Revenue:</strong> Estimated at <strong>$10 million</strong> (including web channels).</li><li><strong>Features:</strong> Offers over <strong>200 voices</strong> in more than <strong>60 languages</strong>, with functionalities like AI Voice Over, AI Video Studio, AI Dubbing, and Voice Cloning.</li><li><strong>Strengths:</strong> Strong brand presence, wide range of voices, extensive feature set.</li><li><strong>Considerations</strong>: Reliance on third-party providers may present challenges in rapidly advancing their technology to keep pace with emerging competitors who develop in-house models.</li></ul><p><a href="https://www.getpeech.com/"><strong>Peech</strong></a></p><ul><li><strong>Market Position:</strong> Emerging player in the TTS market with a focus on user-friendly interfaces and natural-sounding voices.</li><li><strong>Users:</strong> Over <strong>1.5 million downloads</strong> across platforms.</li><li><strong>Features:</strong> Emphasizes high-quality voices, ease of use, and supports multiple languages. Multi-voiced structured narration.</li><li><strong>Innovation</strong>: By incorporating user feedback, especially from fiction lovers, we’ve introduced features that make <a href="https://medium.com/@peechapp/ministry-of-magic-bringing-life-into-fiction-books-with-engaging-audio-54d204da8575">fiction listening more engaging and less tedious</a>.</li><li><strong>Weaknesses:</strong> Smaller market share compared to leading competitors.</li></ul><p><strong>Other Notable Players</strong></p><ul><li><strong>Voice Dream Reader:</strong> Known for extensive customization options, catering to users with specific needs such as dyslexia.</li><li><strong>NaturalReader:</strong> Offers both personal and commercial TTS solutions with a variety of voice options.</li><li><strong>Reader by ElevenLabs:</strong> Provides advanced voice cloning and dubbing services, impacting both B2C and B2B markets. New to B2C with a plan to introduce paid subscriptions soon.</li></ul><h4><strong>Target Audience &amp; Use Cases</strong></h4><p>Understanding our diverse user base is crucial for tailoring our services. Based on data from Peech, we’ve identified primary segments::</p><p><strong>Professionals and Students</strong></p><ul><li><strong>Characteristics:</strong> Individuals seeking to maximize productivity and manage large volumes of reading material efficiently.</li><li><strong>Motivations:</strong> Stay informed, utilize downtime effectively, enhance learning.</li><li><strong>Needs:</strong> Customizable listening experiences, seamless content import, integration features.</li></ul><p><strong>Individuals with Learning Disabilities</strong></p><ul><li><strong>Characteristics:</strong> People with ADHD, <a href="https://medium.com/@peechapp/understanding-dyslexia-how-peech-is-making-a-difference-453ef3fe44b3">dyslexia</a>, or visual impairments who use TTS to improve focus and comprehension.</li><li><strong>Motivations:</strong> Overcome challenges associated with traditional reading, reduce overwhelm.</li><li><strong>Needs:</strong> Features supporting simultaneous reading and listening, tools enhancing concentration and understanding.</li></ul><p><strong>Casual Listeners and Multitaskers</strong></p><ul><li><strong>Characteristics:</strong> Individuals who use TTS for leisure or to multitask, including listening to articles, stories, or documents while engaged in other activities.</li><li><strong>Motivations:</strong> Enhance productivity, enjoy content during downtime.</li><li><strong>Needs:</strong> Easy content import, offline listening capabilities, natural-sounding voices.</li></ul><h4><strong>Market Potential Among Users with Disabilities</strong></h4><p>The potential market size for TTS tools among individuals with disabilities is significant:</p><ul><li><strong>ADHD:</strong> Affects approximately <strong>268 million people globally</strong>. An estimated <strong>20–30%</strong> (~54–80 million) could benefit from TTS.</li><li><strong>Dyslexia:</strong> Impacts between <strong>400 million to 1 billion</strong> individuals. About <strong>20–40%</strong> (~80–400 million) may use TTS for <a href="https://medium.com/@peechapp/understanding-dyslexia-how-peech-is-making-a-difference-453ef3fe44b3">reading assistance</a>.</li><li><strong>Visual Impairments:</strong> Approximately <strong>253 million people</strong> could use TTS, with <strong>30–50%</strong> (~76–127 million) being strong candidates.</li></ul><p>These figures highlight a substantial opportunity to make a meaningful impact while tapping into a vast user base.</p><h3><strong>B2B Market — For Publishers, Educational Institutions and Content Creators</strong></h3><p>Analyzing B2B separately from B2C is essential because the dynamics, sales cycles, and revenue models differ significantly. B2B transactions often involve longer sales cycles, higher deal values, and require building relationships with key stakeholders. In contrast, B2C focuses on high-volume, low-touch sales with shorter decision-making processes.</p><p>The B2B market for Text-to-Speech technology spans several key sectors, each with its own growth trajectory and revenue potential.</p><h4><strong>Educational Sector</strong></h4><p>The global scholarly publishing market is valued at approximately <strong>$28 billion</strong>. Of this, content accessibility (including digital transformation and assistive technologies) is a significant sub-segment. Let’s estimate the TTS opportunity:</p><ul><li><strong>Market Adoption</strong>: If <strong>5–10% of educational institutions</strong> adopt TTS solutions for accessibility and content enhancement, this creates a revenue opportunity of approximately <strong>$1.4-$2.8 billion annually</strong>.</li><li><strong>TTS Revenue per Institution</strong>: Assuming an average annual subscription or licensing fee of <strong>$5,000-$10,000 per institution</strong>, targeting <strong>280,000 institutions globally</strong> could generate <strong>$1.4-$2.8 billion</strong>.</li><li><strong>Content Licensing</strong>: By partnering with publishers for audio-enabled textbooks and materials, TTS providers could tap into additional revenues estimated at <strong>$500 million annually</strong>.</li></ul><h4><strong>Audiobook Production</strong></h4><p>The audiobook market is expected to grow to <strong>$35.04 billion by 2030</strong>, with <strong>26.2% CAGR</strong>. AI-driven TTS technology has the potential to significantly reduce production costs:</p><ul><li><strong>Production Cost Savings</strong>: Traditional audiobook production costs range from <strong>$3,000-$30,000 per book</strong>. TTS solutions could reduce this by <strong>50–70%</strong>, allowing producers to increase the volume of content while maintaining profitability.</li><li><strong>TTS Share of Audiobook Market</strong>: If TTS-generated audiobooks capture <strong>10–20%</strong> of the total market, this translates to <strong>$3.5-$7 billion annually</strong> by 2030.</li></ul><h4><strong>Content Creation and Digital Publishing</strong></h4><p>The global content marketing industry is worth <strong>$487 billion</strong>, with increasing demand for audio content across platforms:</p><ul><li><strong>Adoption of TTS</strong>: If <strong>2–3% of content marketing budgets</strong> shift to TTS solutions for audio-enabled content, this creates a revenue opportunity of approximately <strong>$9.7-$14.6 billion annually</strong>.</li><li><strong>API Revenue</strong>: Companies like Murf AI and PlayAI generate revenue through API integrations, charging between <strong>$0.01-$0.10 per character</strong>. Assuming large-scale use cases across news platforms, publishers, and agencies, API-driven revenues could reach <strong>$1-$2 billion annually</strong>.</li></ul><h4><strong>Customer Service and IVR Systems</strong></h4><p>The global IVR (Interactive Voice Response) market is valued at <strong>$4.3 billion</strong>, with TTS being a core component:</p><ul><li><strong>Market Penetration</strong>: If TTS solutions capture <strong>20–30%</strong> of this market, this translates to an annual revenue potential of <strong>$860 million-$1.29 billion</strong>.</li><li><strong>Per License Revenue</strong>: With enterprise licensing models averaging <strong>$20,000 annually per deployment</strong>, this segment could offer <strong>$500 million-$1 billion</strong> in recurring revenues.</li></ul><h4><strong>Healthcare Applications</strong></h4><p>The healthcare TTS market includes patient education, accessibility, and digital health platforms:</p><ul><li><strong>Market Value</strong>: Estimated at <strong>$2 billion</strong> globally. If TTS solutions capture <strong>15–20%</strong>, this represents <strong>$300-$400 million annually</strong>.</li><li><strong>Per Provider Revenue</strong>: Assuming <strong>50,000 healthcare providers</strong> adopt TTS with annual fees of <strong>$5,000</strong>, this creates an additional revenue stream of <strong>$250 million</strong>.</li></ul><h4><strong>Revenue Breakdown by B2B Segment</strong></h4><pre><br>+-------------------------+--------------+-----------+---------------------+<br>|         Segment         | Market Size  | TTS Share |  Revenue Potential  |<br>+-------------------------+--------------+-----------+---------------------+<br>| Education               | $28 billion  | 5-10%     | $1.4-$2.8 billion   |<br>| Audiobook Production    | $35 billion  | 10-20%    | $3.5-$7 billion     |<br>| Content Creation        | $487 billion | 2-3%      | $9.7-$14.6 billion  |<br>| Customer Service (IVR)  | $4.3 billion | 20-30%    | $860M-$1.29 billion |<br>| Healthcare Applications | $2 billion   | 15-20%    | $300-$400 million   |<br>+-------------------------+--------------+-----------+---------------------+</pre><h3><strong>Conclusion</strong></h3><p>The future of text-to-speech technology is not just promising — it’s transformative. As TTS solutions become more advanced and accessible, they have the power to revolutionize how we consume information, engage with content, and interact with technology.</p><p>The combined revenue potential from these markets — spanning education, audiobooks, content creation, customer service, and healthcare — exceeds <strong>$15–$25 billion annually</strong>. This underscores not only the economic opportunity but also the profound societal impact of making content more accessible and engaging for all.</p><p>As we look ahead, we invite industry stakeholders, content creators, and innovators to join us in shaping the future of text-to-speech technology. Together, we can unlock new possibilities and enhance the way the world communicates.</p><h4><strong>Note About the Author</strong></h4><p><a href="https://www.linkedin.com/in/andreypoznyak/"><strong>Andrey Paznyak</strong></a> is the CEO of Peech, a text-to-speech startup dedicated to transforming the way we consume content with a passion for innovation and a commitment to accessibility.</p><p>By drawing on our insights at Peech and presenting a structured analysis of the market, we hope this article provides valuable perspectives on the dynamic landscape of text-to-speech technology. The opportunities are vast, and the future is undoubtedly exciting for all involved in this transformative field.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89d1aa6ae33a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ministry of Magic: Bringing life into fiction books with engaging audio]]></title>
            <link>https://medium.com/@peechapp/ministry-of-magic-bringing-life-into-fiction-books-with-engaging-audio-54d204da8575?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/54d204da8575</guid>
            <category><![CDATA[audiobooks]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[b2b]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Mon, 01 Jul 2024 12:36:03 GMT</pubDate>
            <atom:updated>2024-12-03T00:08:43.799Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E_RcI800D1JmrOniBBQ-Og.png" /></figure><p>Imagine diving into your favorite book, not just reading but truly experiencing the magic, emotions, and atmosphere as if you were there. This transformation in the way we experience text is at the heart of what we do at <a href="https://www.getpeech.com/">Peech</a>. What started as a <a href="https://apps.apple.com/app/apple-store/id1429704005?pt=119230880&amp;ct=medium&amp;mt=8">simple app</a> to listen to web articles has evolved into a sophisticated platform for AI audio generation. With over a million downloads, our journey has brought us to the forefront of audiobook creation for both individuals and publishers. In this article, we will explore our approach to making audiobooks for publishers and how we got here.</p><h3>Discovering the Magic of Fiction Lovers</h3><p>At Peech, we cater to a variety of user categories. Our users include students who listen to research papers, professionals who keep up with science, finance, and sports articles from well-known sources, and individuals with <a href="https://medium.com/@peechapp/understanding-dyslexia-how-peech-is-making-a-difference-453ef3fe44b3">dyslexia</a> or ADHD who benefit from our text-to-speech technology. Last year, we discovered another enthusiastic group: fiction lovers who consume vast amounts of web novels, detective &amp; romantic stories, and other genres. Some of these users listen to up to 50 books per month! Intrigued by their dedication, we engaged in extensive conversations with them. While they were quite happy with our service, they also highlighted areas for improvement. They wanted more immersive and less monotonous narration, especially considering how long these books are. We quickly set up a process of getting feedback from them by sending new versions of audiobooks privately for assessment. This was our catalyst to innovate and bring stories to life in a whole new way.</p><h3>From Text to Engaging Audio: A Harry Potter Example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GersJ5PRvI4LKwSekC47Vg.jpeg" /></figure><p>To demonstrate how our technology evolved, works, and more importantly, how it sounds, let’s use an example from one of the most beloved fictional series, “<strong>Harry Potter and the Philosopher’s Stone</strong>.” For this article, we’ll use this familiar text to showcase the transformation from static text to an immersive auditory experience.</p><h4><strong>The Original Text</strong></h4><blockquote>Professor McGonagall now stepped forward holding a long roll of parchment. ‘When I call your name, you will put on the hat and sit on the stool to be sorted,’ she said. ‘Abbott, Hannah!’ A pink-faced girl with blonde pigtails stumbled out of line, put on the hat, which fell right down over her eyes, and sat down. A moment’s pause ‘HUFFLEPUFF!’ shouted the hat. The table on the right cheered and clapped as Hannah went to sit down at the Hufflepuff table. Harry saw the ghost of the Fat Friar waving merrily at her. ‘Potter, Harry!’ As Harry stepped forward, whispers suddenly broke out like little hissing fires all over the hall. ‘Potter, did she say?’ ‘The Harry Potter?’ The last thing Harry saw before the hat dropped over his eyes was the Hall full of people craning to get a good look at him. Next second he was looking at the black inside of the hat. He waited. ‘Hmmmm,’ said a small voice in his ear. ‘Difficult. Very difficult. Plenty of courage, I see. Not a bad mind, either. There’s talent, Oh my goodness! Yes! And a nice thirst to prove yourself! Now that’s interesting … So, where shall I put you?’ Harry gripped the edges of the stool and thought, ‘Not Slytherin, not Slytherin.’ ‘Not Slytherin, eh?’ said the small voice. ‘Are you sure? You could be great, you know, it’s all here in your head, and Slytherin will help you on the way to greatness, no doubt about that — no? Well, if you’re sure — better be GRYFFINDOR!’</blockquote><h4>Our Starting Point</h4><p>Initially, our narration was straightforward, with a single voice reading through the text. While functional, it lacked the emotional depth and variety that listeners crave.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1860145536%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fpeech-app%2Fsorting-hat-scene-single-voice%3Fsi%3D4eb08f44bb344ac8a7abc5ba40ed34a9%26utm_source%3Dclipboard%26utm_medium%3Dtext%26utm_campaign%3Dsocial_sharing&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-z5zAqYwqA3VEUIR9-vyVzJQ-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/de5e95ebdaab2600b2a6fbaf06860740/href">https://medium.com/media/de5e95ebdaab2600b2a6fbaf06860740/href</a></iframe><h3><strong>Breathing Life into Characters</strong></h3><p>We started with a simple idea: to captivate our listeners, we needed to assign distinct voices to different characters and infuse the narration with emotion. This required sophisticated language models and natural language processing (NLP) techniques.</p><p>There are two common ways books are written:</p><ul><li><strong>First-Person View:</strong> “I came into the room and saw Ron standing in front of me.”</li><li><strong>Third-Person View:</strong> “Harry came into the room and saw Ron standing in front of him.”</li></ul><p>In the first-person view, a character narrates the story, so we keep their voice consistent in dialogs. In the third-person view, we use a separate voice for the narrator (the author) and different voices for each character in the dialogs.</p><p>All right, here’s a breakdown of the task:</p><ol><li>Check if the text is narrated from a 1st or 3rd person POV.</li><li>Extract all characters involved in dialogs.</li><li>Define the characters’ personalities and relationships (age, gender).</li><li>Assign a dedicated voice to each character based on their personalities.</li><li>Track in the text to whom each line belongs.</li><li>Identify the sentiment of each line (angry, shouting, whispering, friendly, hopeful, etc.)</li></ol><p>This is what we have as a result:</p><ul><li><strong>Professor McGonagall:</strong> A stern yet kind middle-aged female voice.</li><li><strong>The Sorting Hat:</strong> An ancient, deep, and expressive male voice.</li><li><strong>Harry Potter:</strong> A young, hopeful teenage male voice.</li><li><strong>Narrator POV:</strong> Third person, so a dedicated voice for the narrator is needed.</li></ul><p>Our technology automates the assignment of these voices, ensuring each line is delivered with the appropriate emotion, whether it’s excitement, nervousness, or curiosity. The result is a far richer and more engaging narrative. Check it out:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1860149550%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fpeech-app%2Fsorting-hat-scene-multiple-voices%3Fsi%3Df38cc16946a940c2900ca3cfccf27628%26utm_source%3Dclipboard%26utm_medium%3Dtext%26utm_campaign%3Dsocial_sharing&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-z5zAqYwqA3VEUIR9-vyVzJQ-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/27f6ea9285d45aaa3a15bc42f209674a/href">https://medium.com/media/27f6ea9285d45aaa3a15bc42f209674a/href</a></iframe><h3><strong>Adding Magic with Sound Effects</strong></h3><p>We didn’t stop at character voices. To further immerse our listeners, we added short sound effects and ambient sounds.</p><h4><strong>Short Sound Effects</strong></h4><p>These have short and consistent durations, adding dynamic elements to the narration. We categorize them into various groups:</p><ul><li><strong>Animal Sounds:</strong> Roar, bark</li><li><strong>Battle Sounds:</strong> Gunshot, sword clang</li><li><strong>People Sounds:</strong> Breath, heartbeat</li><li><strong>Object Sounds:</strong> Opening and closing doors</li><li><strong>Steps:</strong> Indoor/outdoor running, walking</li><li><strong>Device Sounds:</strong> Mobile notification, phone ring</li><li>… and much more</li></ul><p>To incorporate short sound effects, we only need to know where the sound starts in the text, and we play it once.</p><p>Finding the right effect and creating a library of those was a fun and challenging task. We started with sounds like an old phone ringing, but quickly realized that modern stories required the sounds of new mobile phones. This constant need to extend our library or generate proper sounds on the fly has kept our work dynamic and ever-evolving.</p><p>Here is the next version of our audio, enriched with <strong>footsteps</strong>, <strong>applause</strong>, and even the <strong>sound of a fast-beating heart</strong> to reflect Harry’s anxiety:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1860202113%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fpeech-app%2Fsorting-hat-scene-short-sound-effects-cut%3Fsi%3D256b5d2532d543119297d5d4d51944e7%26utm_source%3Dclipboard%26utm_medium%3Dtext%26utm_campaign%3Dsocial_sharing&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-z5zAqYwqA3VEUIR9-vyVzJQ-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa7dea785ef2a10095176846a8d97351/href">https://medium.com/media/fa7dea785ef2a10095176846a8d97351/href</a></iframe><p>Full scene:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1860162978%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fpeech-app%2Fsorting-hat-scene-short-sound-effects%3Fsi%3D53f240545479445ea677f4958b8ecaa5%26utm_source%3Dclipboard%26utm_medium%3Dtext%26utm_campaign%3Dsocial_sharing&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-z5zAqYwqA3VEUIR9-vyVzJQ-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/38c5cf034be6e4ede2bd82c89aaff408/href">https://medium.com/media/38c5cf034be6e4ede2bd82c89aaff408/href</a></iframe><h4><strong>Ambient Sounds</strong></h4><p>For longer scenes, we incorporated ambient sounds like weather effects and background music to set the mood and create a sense of place. Apart from matching the context, the technical challenge lies in determining the duration, starting and ending points, and mixing these sounds with the short sound effects and main narration seamlessly.</p><p>Result with ambient that accompanies the theme and sentiment:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1860163701%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fpeech-app%2Fsorting-hat-scene-sound-effects-ambients%3Fsi%3D2bb4f055bd424667996c121faaae6d85%26utm_source%3Dclipboard%26utm_medium%3Dtext%26utm_campaign%3Dsocial_sharing&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-z5zAqYwqA3VEUIR9-vyVzJQ-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/28f55f69fb721e5018864c9eaf3ee4a3/href">https://medium.com/media/28f55f69fb721e5018864c9eaf3ee4a3/href</a></iframe><h3><strong>The Power of Advanced Technology</strong></h3><p>Under the hood, there’s a lot of heavy tech stuff happening: advanced Large Language Models (LLMs), Natural Language Processing (NLP), and even in some cases old-school regular expressions to manage the tasks outlined above. Our<a href="https://medium.com/@peechapp/text-to-speech-models-part-1-intro-little-theory-and-math-0ffa5d3e0e3f"> own AI synthesis</a> plays a significant role in creating high-quality audio.</p><p>However, this year some companies introduced completely new models for AI voice generation that are extremely hard to catch up with in terms of quality and intonation. These advancements can, in some cases, even enhance the end result.</p><p>So here is the same audio but narrated with <a href="https://elevenlabs.io/">ElevenLabs</a> voices:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F1860164322%26show_artwork%3Dtrue&amp;display_name=SoundCloud&amp;url=https%3A%2F%2Fsoundcloud.com%2Fpeech-app%2Fsorting-hat-scene-complete-elevenlabs-voices%3Fsi%3D9511d342479c41aba722fed3974f639d%26utm_source%3Dclipboard%26utm_medium%3Dtext%26utm_campaign%3Dsocial_sharing&amp;image=https%3A%2F%2Fi1.sndcdn.com%2Fartworks-z5zAqYwqA3VEUIR9-vyVzJQ-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ff313d453f6e1ec81276867a999e288/href">https://medium.com/media/6ff313d453f6e1ec81276867a999e288/href</a></iframe><h3><strong>The Shift to B2B</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*5Qm2xevqNw6zlXaKsE32Sw.jpeg" /><figcaption>b2b — startup — b2c</figcaption></figure><p>Our journey into B2B was driven by a simple realization: we had developed quite unique solution for making audiobooks. However, the processing required for these advanced features is too intensive for real-time applications on our iOS app. Thus, we turned to the B2B market, where we could deliver high-quality, pre-processed audiobooks directly to the copyright holders.</p><p>This decision quickly bore fruit. We secured contracts with companies providing web novel and fiction reading services, offering them a cost-effective way to transform their content into professional-grade audiobooks. These partnerships allowed even amateur authors to bring their stories to life in an engaging format, reaching a wider audience and enhancing their reader’s experience.</p><p>By focusing on B2B, we could ensure that our technology had the time and resources needed to perform at its best, without the constraints of real-time processing. This move has opened up new opportunities and marked a significant milestone in our company’s journey. In fact, we have already produced over 30,000 hours of audiobooks, at a cost and speed that was previously unimaginable. Complex audiobooks that once took weeks to produce can now be generated in just a few hours.</p><h3><strong>About the Authors</strong></h3><p>Peech was co-founded by <a href="https://www.linkedin.com/in/aliaksei-horbel-5b42a1113/">Aliaksei Horbel</a> and <a href="https://www.linkedin.com/in/andreypoznyak/">Andrey Paznyak</a> in 2020, who saw the potential for turning text into audio experiences for individuals. Both with technical background, Andrey, the CEO, oversees the company’s operations, while Aliaksei leads the B2B direction and is the main architect behind the innovative solutions described here.</p><h3><strong>Conclusion</strong></h3><p>Our foray into B2B has not only allowed us to showcase the true potential of our technology but also to bring stories to life in ways that were previously unimaginable. As we continue to innovate and refine our offerings, we look forward to sharing more about our journey and the magic we’re creating in the world of audiobooks.</p><p>There are quite a lot of technical challenges in every step of the process described above so we’ll keep the details for another articles. Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54d204da8575" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Dyslexia: How Peech is Making a Difference]]></title>
            <link>https://medium.com/@peechapp/understanding-dyslexia-how-peech-is-making-a-difference-453ef3fe44b3?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/453ef3fe44b3</guid>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[dyslexia]]></category>
            <category><![CDATA[inclusivity]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[ios-app-development]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Thu, 20 Jun 2024 18:37:58 GMT</pubDate>
            <atom:updated>2024-12-03T00:01:44.166Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8gPb0fi5vtbttMGPF0hsjw.png" /><figcaption>Listening &amp; Reading</figcaption></figure><p>At <a href="https://www.getpeech.com/">Peech</a>, we’re on a mission to make reading a breeze for everyone. As tech enthusiasts from Eastern Europe, we initially had no clue about dyslexia when we launched our text-to-speech AI startup. We simply wanted to let us and users listen to web articles, ebooks, and PDFs on their iOS devices — think of it as giving your eyes a break while your ears do the work! Over time, we also ventured into the B2B space, helping publishers transform their content into audiobooks.</p><p>But then as we progressed something interesting happened. We started noticing a significant number of our users with good retention rates had a common trait — as we dug deeper we discovered that many of them had dyslexia. After conducting user interviews, we started to explore how Peech could make a significant impact on the lives of people with dyslexia. So, sit tight as we dive into what dyslexia is, the challenges it poses, and how Peech is stepping up to help.</p><h3>What is Dyslexia?</h3><p>Dyslexia is a learning disorder characterized by difficulties with reading, writing, and spelling. It’s not related to intelligence; rather, it’s a difference in how the brain processes written and spoken language. According to the <a href="https://dyslexiaida.org/">International Dyslexia Association</a>, about 10% of the population has dyslexia, but in many regions, including ours, awareness and support are limited.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fgv-af_Pqd4FxMcrHRb51A.jpeg" /><figcaption>Dyslexia Illustration generated by DALLE</figcaption></figure><h3>Challenges Faced by People with Dyslexia</h3><p>People with dyslexia often struggle with:</p><ul><li><strong>Decoding Words</strong>: Difficulty breaking down words into sounds.</li><li><strong>Reading Fluency</strong>: Slow and laborious reading.</li><li><strong>Spelling</strong>: Persistent spelling errors.</li><li><strong>Comprehension</strong>: Trouble understanding and remembering what they read.</li></ul><p>But dyslexia can also affect more than just reading. Many people with dyslexia face challenges with:</p><ul><li><strong>Motor Skills</strong>: Tasks requiring fine motor skills, such as tying shoelaces, can be difficult. For example, actor Daniel Radcliffe <a href="https://www.theunwritten.co.uk/2022/01/10/how-the-media-discusses-daniel-radcliffes-dyspraxia-is-outdated-and-damaging/">has mentioned </a>that he struggles with motor skills due to his <a href="https://en.wikipedia.org/wiki/Developmental_coordination_disorder">dyspraxia</a>, which often co-occurs with dyslexia.</li><li><strong>Time Management and Organization</strong>: Keeping track of time and organizing tasks can be challenging.</li></ul><p>These challenges can affect academic performance, self-esteem, and overall quality of life. Still, in many countries, including some of our favorite in Eastern Europe 😉, dyslexia is not widely recognized, leading to a lack of resources and support for those who need it.</p><h3>Famous People with Dyslexia</h3><p>Dyslexia is more common than you might think, and many successful individuals have dyslexia. Here are a few inspiring examples:</p><ul><li><strong>Albert Einstein</strong>: The legendary physicist who developed the theory of relativity <a href="https://journal.imse.com/12-famous-people-who-struggled-with-dyslexia-before-changing-the-world/">struggled</a> with reading and spelling as a child.</li><li><strong>Richard Branson</strong>: The founder of Virgin Group <a href="https://www.virgin.com/branson-family/richard-branson-blog/my-message-to-anyone-disheartened-by-exam-results">often speaks about</a> his dyslexia and how it shaped his entrepreneurial spirit.</li><li><strong>Steven Spielberg</strong>: The acclaimed filmmaker <a href="https://childmind.org/blog/steven-spielberg-talks-bullying-and-dyslexia/">has discussed how his dyslexia</a> was diagnosed later in life and how it influenced his work.</li><li><strong>Bill Gates</strong>: The co-founder of Microsoft <a href="https://elitebusinessmagazine.co.uk/people/item/dyslexia-in-the-workplace-stigma-or-superpower">has spoken about his dyslexia</a> and how he turned his unique way of thinking into a strength.</li></ul><h3>The Importance of Inclusivity in Education</h3><p>Dyslexia affects a significant portion of the student population. It’s estimated that around 10–15% of students in schools and universities have dyslexia. This means millions of students face daily challenges in their academic pursuits. Without proper support, these students can struggle with reading, writing, and comprehension, which can lead to lower academic performance and self-esteem.</p><p>Obviously, unlike us, this is not breaking news for some of the governments, so there is already quite a strong background and actions taken to address this issue.</p><p><strong>Global Efforts to Support Students with Dyslexia:</strong></p><ul><li><strong>United States</strong>: The Individuals with Disabilities Education Act (<a href="https://sites.ed.gov/idea/">IDEA</a>) ensures that students with dyslexia receive accommodations such as extended test times, audiobooks, and individualized education plans (IEPs).</li><li><strong>United Kingdom</strong>: <a href="https://dyslexiaaction.org.uk/">The Dyslexia Action organization</a> provides resources and support for students, including specialized tutoring and assistive technology.</li><li><strong>Australia</strong>: The government funds programs to support students with learning difficulties, including dyslexia, through resources like the <a href="https://www.ndis.gov.au/">National Disability Insurance Scheme</a> (NDIS).</li><li><strong>Finland</strong>: Known for its inclusive education system, Finland <a href="https://www.oph.fi/en/education-system">provides</a> comprehensive support for dyslexic students, including specialized teachers and tailored learning plans.</li></ul><h3>How Peech is Helping</h3><p>After studying this interesting condition and, of course, talking to our users, we implemented two key features: spoken text highlight and dyslexia-friendly fonts. Both of these features complement our main offering, which is listening to texts.</p><h3>Spoken Text Highlight</h3><p>This feature visually highlights words as they are read aloud, providing a synchronized audio and visual experience. It helps users follow along with the text, improving focus and comprehension. Technically, developing this feature was challenging, and soon we will post a tech article showcasing its implementation in terms of iOS / Swift and Cloud server-side.</p><p>This is how it looks on the app itself:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FJOToVAmLgoI&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DJOToVAmLgoI&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FJOToVAmLgoI%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=google" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/eadeae9e31bc22c6b96f8d155799ada0/href">https://medium.com/media/eadeae9e31bc22c6b96f8d155799ada0/href</a></iframe><p><a href="https://youtube.com/shorts/Y5KrQYaW8ng?feature=share">Peech Spoken Text Highlight &amp; Dyslexia Font Demo</a></p><h3>Dyslexia-Friendly Fonts</h3><p>In our latest release, we added the option to switch to dyslexia-friendly fonts like OpenDyslexic. This font is designed to improve readability and reduce errors.</p><p><strong>How Dyslexia-Friendly Fonts Help:</strong></p><ul><li><strong>Distinctive Shapes</strong>: Each letter has a unique shape, making it easier to distinguish.</li><li><strong>Weighted Bottoms</strong>: The weighted bottoms of letters help prevent them from being flipped or reversed.</li><li><strong>Larger Spaces</strong>: Increased spacing between letters and words helps reduce visual crowding.</li></ul><p>This is how it looks on the app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*3z3c7FiYNqlZPZ-BXJ_yWg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*-M_JpLuUXEhHackXZzCmHQ.jpeg" /></figure><h3>About the Author</h3><p>Hi, I’m <a href="https://www.linkedin.com/in/andreypoznyak/">Andrey Paznyak</a>, CEO of Peech. I have a huge technical background, which is why I sometimes write about programming. I don’t have dyslexia myself, so I wasn’t aware of it when we first launched Peech. However, discovering the significant number of our users with dyslexia has been an eye-opening experience and has driven us to make Peech more accessible for everyone. See below on how to integrate dyslexia-friendly fonts into an iOS app.</p><h3>Technical Insights: Adding OpenDyslexic font to Your App in Swift</h3><p>As a developer, you might be interested in how we integrated the OpenDyslexic font into our app. Here’s a quick guide:</p><ol><li><strong>Download the Font</strong>:<br>You can download the OpenDyslexic font from<a href="https://opendyslexic.org/"> this link</a>. Don’t forget to support the creators with a small contribution 😊</li><li><strong>Add the Font to Your Project:</strong><br> — Drag and drop the font files into your Xcode project.<br> — Ensure that the font files are included in your project’s bundle.</li><li><strong>Update the <em>Info.plist</em></strong>:<br> — Add a new entry for the fonts under `UIAppFonts` in your Info.plist file:</li></ol><pre>&lt;key&gt;UIAppFonts&lt;/key&gt;<br> &lt;array&gt;<br> &lt;string&gt;OpenDyslexic-Regular.otf&lt;/string&gt;<br> &lt;string&gt;OpenDyslexic-Bold.otf&lt;/string&gt;<br>&lt;/array&gt;</pre><p>4. <strong>Apply the Font in Your App</strong>:<br> — Use the font in your UILabels, UITextViews or within your AttributedStrings, etc.:</p><pre>let font = UIFont(name: &quot;OpenDyslexic-Regular&quot;, size: 17)<br><br>label1.font = font<br><br>label2.attributedText = NSAttributedString(string: &quot;Hello world&quot;, attributes: [<br>   .font: font,<br>   .foregroundColor: UIColor.black100Dynamic,<br>])</pre><p>5. <strong>Don’t forget to add fallback fonts in case the alphabet is out of range</strong>.</p><p>By the way, apart from OpenDyslexic fonts family, the community also appreciates more classic Lexend and Arial ones. Here’s how to make it work in Swift:</p><pre>guard let primaryFont = UIFont(name: &quot;OpenDyslexic-Regular&quot;, size: size) else { return nil }<br><br>let primaryFontDescriptor = primaryFont.fontDescriptor<br>let fallbackFontDescriptors = [&quot;Lexend-Regular&quot;, &quot;Arial-Regular&quot;].map { UIFontDescriptor(name: $0, size: size) }<br>        <br>let combinedFontDescriptor = primaryFontDescriptor.addingAttributes([<br>    .cascadeList: fallbackFontDescriptors<br>])<br><br>return UIFont(descriptor: combinedFontDescriptor, size: size)</pre><p>By following these steps, you can easily integrate dyslexia-friendly fonts into your own applications, making them more accessible.</p><h3>Conclusion</h3><p>At Peech, we’re committed to making reading accessible to everyone. By understanding the challenges faced by people with dyslexia and implementing thoughtful features, we hope to make a positive impact. Whether you’re a user or a fellow developer, we invite you to join us in creating a more inclusive digital world. Feel free to drop us a message at product@getpeech.com. That’s all for today 😃</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=453ef3fe44b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text-To-Speech Models. Part 6: DelightfulTTS implementation and training]]></title>
            <link>https://medium.com/@peechapp/text-to-speech-models-part-6-delightfultts-implementation-and-training-ceb4c94fd5fb?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ceb4c94fd5fb</guid>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[speech]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 09:21:52 GMT</pubDate>
            <atom:updated>2024-06-11T14:59:23.635Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Tweaking, Implementing and Training DelightfulTTS Model</strong></h3><h3>Read the Full Series of ML TTS exploration</h3><ul><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-1-intro-little-theory-and-math-0ffa5d3e0e3f">Part 1: Intro, Little Theory and Math 📘</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-2-autoregressive-models-world-636d8aa0932d">Part 2: Autoregressive models world 🌍</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-3-non-autoregressive-models-hideout-f45e46a4f4bd">Part 3: Non-autoregressive models hideout 🕵️‍♂️</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-4-autoregressive-and-hybrid-models-review-abeb2b5ec3ec">Part 4: Autoregressive and hybrid models review 📊</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-5-non-autoregressive-models-review-7ce8de9c9e9d">Part 5: Non-autoregressive models review📝</a></li><li>👉Part 6: DelightfulTTS implementation and training 🛠️</li></ul><p><em>Everything we are covering in this article can be found in 🔗</em><a href="https://github.com/PeechApp/tts-peech"><em>TTS-Peech github repo</em></a><em>. The repository serves as a comprehensive resource where you can explore the implementation details, review the documentation, and examine the test cases that ensure the code’s reliability and correctness. You can find all the documentation inside the </em><strong><em>docs</em></strong><em> directory, run the docs locally with </em><strong><em>mkdocs serve. </em></strong><em>Also here you can take a look on the </em><a href="https://peechapp.github.io/tts-peech/"><em>docs online</em></a><em>.</em></p><h3>Intro</h3><p>On the way to building state-of-the-art text-to-speech and after <a href="https://medium.com/@peechapp/text-to-speech-models-part-5-non-autoregressive-models-review-7ce8de9c9e9d">conducting research</a>, I decided to implement the DelightfulTTS model. I didn’t find many details in the paper about the implementation and hyperparameters, but I found the <a href="https://github.com/keonlee9420/Comprehensive-Transformer-TTS">Comprehensive-Transformer-TTS</a> repository and was heavily influenced by this implementation. For example, you can check the Conformer implementation: <a href="https://github.com/keonlee9420/Comprehensive-Transformer-TTS/blob/main/model/transformers/conformer.py">Comprehensive-Transformer-TTS/model/transformers/conformer.py</a></p><p>I also found another great implementation on GitHub: <a href="https://github.com/dunky11/voicesmith">dunky11/voicesmith</a></p><blockquote>VoiceSmith makes it possible to train and infer on both single and multispeaker models without any coding experience. It fine-tunes a pretty solid text to speech pipeline based on a modified version of <a href="https://arxiv.org/abs/2110.12612">DelightfulTTS</a> and <a href="https://arxiv.org/abs/2106.07889">UnivNet</a> on your dataset.</blockquote><p>And there’s also a coqui-ai <a href="https://github.com/coqui-ai/TTS/pull/2095">PR Delightful-TTS</a> implemetation.</p><h3><strong>Pulp Friction</strong></h3><p>I chose to use my own training code due to the flexibility it offers in the training process. While PyTorch Lightning is an excellent training framework that simplifies multi-GPU training and incorporates many best practices, custom implementations come with their own set of advantages and disadvantages.</p><h4><strong>How to Train Our Model</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*vg_3LVqwd7kyLjzoybNEug.jpeg" /></figure><p>Using PyTorch Lightning to train a model is very straightforward. My training script allows you to train on a single machine with either one or multiple GPUs.</p><p>I tried to support the hyperparameters and configurations in one place. You can find the configuration in the <strong>models/config</strong> directory. <strong>PreprocessingConfig</strong> is the basic piece of configuration where you can set the configuration for preprocessing audio used for training.</p><p><strong>PreprocessingConfigUnivNet</strong> and <strong>PreprocessingConfigHifiGAN</strong> have different <strong>n_mel_channels</strong>, and these two configurations are not compatible because this parameter fundamentally defines the architecture of the <strong>AcousticModel</strong>.</p><p>For the <strong>PreprocessingConfigHifiGAN</strong> , we have sampling rates of 22050 or 44100. You can see the post-initialization action that sets the parameters of <strong>stft</strong> based on this parameter:</p><pre>def __post_init__(self):<br> r&quot;&quot;&quot;It modifies the &#39;stft&#39; attribute based on the &#39;sampling_rate&#39; attribute.<br> If &#39;sampling_rate&#39; is 44100, &#39;stft&#39; is set with specific values for this rate.<br> If &#39;sampling_rate&#39; is not 22050 or 44100, a ValueError is raised.<br><br> Raises:<br>  ValueError: If &#39;sampling_rate&#39; is not 22050 or 44100.<br> &quot;&quot;&quot;<br> if self.sampling_rate == 44100:<br>  self.stft = STFTConfig(<br>   filter_length=2048,<br>   hop_length=512,<br>   win_length=2048,<br>   n_mel_channels=80,<br>   mel_fmin=20,<br>   mel_fmax=11025,<br>  )<br> if self.sampling_rate not in [22050, 44100]:<br>  raise ValueError(&quot;Sampling rate must be 22050 or 44100&quot;)</pre><p>To set up the cluster, you need to define the following environment variables:</p><pre># Node runk in the cluster<br>node_rank = 0<br># Num nodes in the cluster<br>num_nodes = 1<br><br>os.environ[&quot;WORLD_SIZE&quot;] = f&quot;{num_nodes}&quot;<br>os.environ[&quot;NODE_RANK&quot;] = f&quot;{node_rank}&quot;<br><br># Setup of the training cluster port<br>os.environ[&quot;MASTER_PORT&quot;] = &quot;12355&quot;<br><br># Change the IP address to the IP address of the master node<br>os.environ[&quot;MASTER_ADDR&quot;] = &quot;10.148.0.6&quot;</pre><p>You can also choose the logs directory and checkpoints paths:</p><pre>default_root_dir = &quot;logs&quot;<br>ckpt_acoustic = &quot;./checkpoints/epoch=301-step=124630.ckpt&quot;<br>ckpt_vocoder = &quot;./checkpoints/vocoder.ckpt&quot;</pre><p>Here’s my Trainer code. I use the <strong>DDPStrategy </strong>because it’s the only strategy that works in my case. Other useful parameters are:</p><pre>trainer = Trainer(<br>    accelerator=&quot;cuda&quot;,  # Use GPU acceleration<br>    devices=-1,  # Use all available GPUs<br>    num_nodes=num_nodes,<br>    strategy=DDPStrategy(<br>        gradient_as_bucket_view=True,  # Optimize gradient communication<br>        find_unused_parameters=True,  # Ignore unused parameters<br>    ),<br>    default_root_dir=default_root_dir,  # Directory for logs<br>    enable_checkpointing=True,  # Enable checkpoint saving<br>    accumulate_grad_batches=5,  # Accumulate gradients for 5 batches<br>    max_epochs=-1,  # Train indefinitely (until manually stopped)<br>    log_every_n_steps=10,  # Log every 10 steps<br>    gradient_clip_val=0.5,  # Clip gradients to a maximum value of 0.5<br>)</pre><p>Import the preprocessing configuration that you want to train and choose the type of training. For example:</p><pre>from models.config import PreprocessingConfigUnivNet as PreprocessingConfig<br><br># ...<br><br>preprocessing_config = PreprocessingConfig(&quot;multilingual&quot;)<br>model = DelightfulTTS(preprocessing_config)<br><br>tuner = Tuner(trainer)<br><br>train_dataloader = model.train_dataloader(<br> # NOTE: Preload the cached dataset into the RAM<br> # Use only if you have enough RAM to cache your entire dataset<br> cache_dir=&quot;/dev/shm/&quot;,<br> cache=True,<br>)<br><br>trainer.fit(<br> model=model,<br> train_dataloaders=train_dataloader,<br> # Resume training states from the checkpoint file<br> ckpt_path=ckpt_acoustic,<br>)</pre><p>The entry point to the project is inside the <strong>train.py</strong> file, which serves as the foundation of the training process.</p><h3><strong>Inference code</strong></h3><p>For inference, you can use the code examples from <strong>demo/demo.py</strong>. You have three possible options for combining the acoustic model and vocoder:</p><ul><li>DelightfulTTS + UnivNet, with a sample rate of 22050 Hz</li><li>DelightfulTTS + HiFi-GAN, with a sample rate of 44100 Hz</li><li>FastPitch + HiFi-GAN, with a sample rate of 44100 Hz</li></ul><p>You can experiment with these TTS models and find the best fit for your use case.</p><h3><strong>Dataset code</strong></h3><p>For managing datasets, I utilize the <a href="https://github.com/lhotse-speech/lhotse"><strong>lhotse</strong></a><strong> </strong>library. This library allows you to wrap your audio dataset, prepare cutset files, and then filter or select specific subsets of the data. For example:</p><pre># Duration lambda<br>self.dur_filter = (<br> lambda duration: duration &gt;= self.min_seconds<br> and duration &lt;= self.max_seconds<br>)<br><br># Filter the HiFiTTS cutset to only include the selected speakers<br>self.cutset_hifi = self.cutset_hifi.filter(<br> lambda cut: isinstance(cut, MonoCut)<br> and str(cut.supervisions[0].speaker) in self.selected_speakers_hi_fi_ids_<br> and self.dur_filter(cut.duration),<br>).to_eager()</pre><p>With lhotse, you can filter your dataset based on various criteria, such as speaker identities or audio duration. Notably, you don’t need to read the audio metadata for this step, as the metadata is already present in the cutset file, which can be prepared beforehand. The lhotse library provides a wide range of functions and utilities for managing and processing speech datasets efficiently. I highly recommend exploring its capabilities to streamline your dataset preparation and filtering tasks.</p><p>I have prepared the following ready-to-use datasets:</p><ol><li><strong>HifiLibriDataset</strong>: This dataset code combines the <a href="http://openslr.org/109/">Hi-Fi Multi-Speaker English TTS Dataset</a> and <a href="http://openslr.org/141/">LibriTTS-R</a>. You can choose specific speakers from these datasets or filter the audio based on various parameters. The dataset code can cache data to the filesystem or RAM if needed. This dataset can be used to train the DelightfulTTS model.</li><li><strong>HifiGanDataset</strong>: This dataset code is designed for the HiFi-GAN model. The data is prepared in a specific way, as you can see in the <strong>__getitem__</strong> method. With this dataset implementation, you can train the HiFi-GAN vocoder.</li><li><strong>LIBRITTS_R</strong>: This code is a modified version of the <a href="https://pytorch.org/audio/main/generated/torchaudio.datasets.LIBRITTS.html">Pytorch LIBRITTS</a> dataset.</li></ol><p>Additionally, you can explore the dataset code I prepared for various experiments within this project.</p><h3><strong>Acoustic model</strong></h3><p>The acoustic model is responsible for generating mel-spectrograms from input phoneme sequences, speaker identities, and language identities.</p><p>The core of your acoustic model is the <strong>AcousticModel </strong>class, which inherits from PyTorch’s <strong>nn.Module</strong>. This class contains several sub-modules and components that work together to generate the mel-spectrogram output.</p><p><strong>Encoder</strong>: The encoder is a Conformer module that processes the input phoneme sequences. It takes the phoneme embeddings, speaker embeddings, language embeddings, and positional encodings as input and generates a hidden representation.</p><p><strong>Prosody Modeling</strong>:</p><ul><li>Utterance-Level Prosody: The model includes an <strong>UtteranceLevelProsodyEncoder</strong> and a <strong>PhonemeProsodyPredictor</strong> (for utterance-level prosody) to capture and predict the prosodic features at the utterance level.</li><li>Phoneme-Level Prosody: Similarly, there is a <strong>PhonemeLevelProsodyEncoder</strong> and a <strong>PhonemeProsodyPredictor</strong> (for phoneme-level prosody) to model the prosodic features at the phoneme level.</li></ul><p><strong>Variance Adaptor:</strong></p><ul><li>Pitch Adaptor: The <strong>PitchAdaptorConv</strong> module is responsible for adding pitch embeddings to the encoder output, based on the predicted or target pitch values.</li><li>Energy Adaptor: The <strong>EnergyAdaptor</strong> module adds energy embeddings to the encoder output, based on the predicted or target energy values.</li><li>Length Adaptor: The <strong>LengthAdaptor</strong> module upsamples the encoder output to match the length of the target mel-spectrogram, using the predicted or target duration values.</li></ul><p><strong>Decoder:</strong> The decoder is another Conformer module that takes the output from the variance adaptor modules and generates the final mel-spectrogram prediction.</p><p><strong>Embeddings:</strong> The model includes learnable embeddings for phonemes, speakers, and languages, which are combined and used as input to the encoder.</p><p><strong>Aligner:</strong> The Aligner module is used during training to compute the attention logits and alignments between the encoder output and the target mel-spectrogram.</p><p>The <strong>forward_train</strong> method defines the forward pass during training, where it takes the input phoneme sequences, speaker identities, language identities, mel-spectrograms, pitches, energies, and attention priors (if available). It computes the mel-spectrogram prediction, as well as the predicted pitch, energy, and duration values, which are used for computing the training loss. The forward method defines the forward pass during inference, where it takes the input phoneme sequences, speaker identities, language identities, and duration control value, and generates the mel-spectrogram prediction. Additionally, there are methods for freezing and unfreezing the model parameters, as well as preparing the model for export by removing unnecessary components (e.g., prosody encoders) that are not needed during inference. Implementation follows the DelightfulTTS architecture and incorporates various components for modeling prosody, pitch, energy, and duration, while leveraging the Conformer modules for the encoder and decoder.</p><h3><strong>Results</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVqiu1rqFejP4ydx2W2_aw.png" /><figcaption><strong>total loss</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jqjRXD1CFEGTJlm7eDAh1w.png" /><figcaption><strong>mel loss</strong></figcaption></figure><p>Check the results:</p><ul><li><a href="https://huggingface.co/spaces/peechapp/PeechTTSv22050">PeechTTS 22050 hz</a></li><li><a href="https://huggingface.co/spaces/peechapp/PeechTTSv44100">PeechTTS 44100 hz</a></li></ul><p>Training the DelightfulTTS model is a computationally intensive task due to its large size and numerous components. Achieving stable and consistent results requires significant computational resources and time. In my experience, using 8 Nvidia V100 GPUs resulted in extremely slow progress, with visible improvements taking weeks to manifest. However, upgrading to 8 Nvidia A100 GPUs significantly accelerated the training process, allowing visible progress to be observed within days. For optimal performance and efficiency, the ideal hardware configuration would be a cluster with 16 Nvidia A100 GPUs. The computational demands of the DelightfulTTS model highlight the importance of leveraging state-of-the-art hardware accelerators, such as the Nvidia A100 GPUs, to enable practical training times and facilitate the development of high-quality text-to-speech systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*FVZ4a-Z2bNpVvN5zNBnKCg.jpeg" /></figure><h3><strong>Future work</strong></h3><p>The combination of autoregressive (AR) and non-autoregressive (NAR) architectures is a promising direction in the text-to-speech (TTS) field. Diffusion models can introduce sufficient variation, and the approach taken by StyledTTS2 is particularly appealing. A potential avenue for future exploration could be to integrate the FastPitch model, known for its speed and stability, with the diffusion block from StyledTTS2. This hybrid approach could leverage the strengths of both models. The NAR nature of FastPitch would contribute to the overall speed and stability of the system, while the diffusion-based component from StyledTTS2 could introduce diversity and expressiveness to the generated speech. By combining the efficient and robust FastPitch model with the expressive capabilities of the diffusion-based variator from StyledTTS2, this hybrid architecture could potentially deliver diverse and natural-sounding speech while maintaining computational efficiency and stability. However, integrating these two distinct approaches into a unified model may pose challenges, and further research would be necessary to investigate the feasibility and effectiveness of such a hybrid solution.</p><h3>Thank you!</h3><p>Author: <a href="https://www.linkedin.com/in/nick-ovchinnikov/">Nick Ovchinnikov</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZzElAoudhbsVOnJOxFHweg.jpeg" /><figcaption>Fin</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ceb4c94fd5fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text-To-Speech Models. Part 5: Non-autoregressive models review]]></title>
            <link>https://medium.com/@peechapp/text-to-speech-models-part-5-non-autoregressive-models-review-7ce8de9c9e9d?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ce8de9c9e9d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[speech]]></category>
            <category><![CDATA[research-and-development]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 09:18:34 GMT</pubDate>
            <atom:updated>2024-06-11T14:59:42.183Z</atom:updated>
            <content:encoded><![CDATA[<h3>Non-Autoregressive Open-Source TTS Models Review</h3><h3>Read the Full Series of ML TTS exploration</h3><ul><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-1-intro-little-theory-and-math-0ffa5d3e0e3f">Part 1: Intro, Little Theory and Math 📘</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-2-autoregressive-models-world-636d8aa0932d">Part 2: Autoregressive models world 🌍</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-3-non-autoregressive-models-hideout-f45e46a4f4bd">Part 3: Non-autoregressive models hideout 🕵️‍♂️</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-4-autoregressive-and-hybrid-models-review-abeb2b5ec3ec">Part 4: Autoregressive and hybrid models review 📊</a></li><li>👉Part 5: Non-Autoregressive models review📝</li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-6-delightfultts-implementation-and-training-ceb4c94fd5fb">Part 6: DelightfulTTS implementation and training 🛠️</a></li></ul><h3>Non AR models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*HZSgv7w7eRy8QE-LPrtKTQ.png" /><figcaption>Typical NAR model</figcaption></figure><p>In the following sections, I will review the most interesting implementations and key ideas in the field of non-autoregressive (NAR) text-to-speech models. It’s important to note that not all models have published research papers or open-source code available. However, I will endeavor to uncover and present the relevant information and insights from the available sources.</p><h3>FastPitch: Parallel Text-to-Speech with Pitch Prediction</h3><p>Implementation <a href="https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_MixerTTS_Training.ipynb">NVIDIA/NeMo: FastPitch and Mixer-TTS Training</a></p><p>Traditional autoregressive TTS models, which generate speech one step at a time, suffer from slow inference speeds, making them unsuitable for real-time applications. FastPitch, proposed by NVIDIA in 2020, is a fully-parallel text-to-speech model that addresses these challenges by introducing parallel pitch prediction and conditioning the mel-spectrogram generation on the predicted pitch contours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*S87moWXEu-IkRsWPvZXuTQ.png" /><figcaption>FastPeech schema</figcaption></figure><p>It is based on FastSpeech and composed mainly of two feed-forward Transformer (FFTr) stacks. The first one operates in the resolution of input tokens, the second one in the resolution of the output frames.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/399f34130440cade82cd1955f3deb414/href">https://medium.com/media/399f34130440cade82cd1955f3deb414/href</a></iframe><p><strong>Key Contributions</strong></p><p><strong>Parallel Pitch Prediction</strong>: One of the main innovations of FastPitch is its ability to predict the fundamental frequency (<strong>F0</strong>) contours, which represent the pitch variations in speech, directly from the input text in a fully-parallel manner. Unlike previous TTS models that predicted pitch contours autoregressively, FastPitch employs a feed-forward architecture (<strong>FFTr</strong>) and a dedicated pitch predictor to generate the pitch contours in a single pass. This parallel pitch prediction eliminates the need for autoregressive processing, leading to faster inference times.</p><p><strong>Conditioning on Pitch Contours</strong>: FastPitch conditions the mel-spectrogram generation, which represents the spectral characteristics of speech, on the predicted pitch contours. By explicitly modeling and incorporating the pitch information, FastPitch can capture and reproduce the natural variations in pitch and intonation present in human speech, resulting in more natural-sounding synthesized audio.</p><p><strong>Feed-Forward Architecture</strong>: FastPitch is a feed-forward model, which means that it does not rely on recurrent neural networks (RNNs) or attention mechanisms. This architecture choice contributes to the model’s efficiency and parallelizability.</p><p><strong>Improved Speech Quality</strong>: By explicitly modeling pitch contours and conditioning the mel-spectrogram generation on them, FastPitch achieves improved speech quality compared to its predecessor, FastSpeech. The synthesized speech exhibits more natural-sounding pitch variations and intonation patterns.</p><p><strong>Limitations and Future Work</strong></p><p>While FastPitch represents a significant advancement in parallel text-to-speech synthesis, the authors acknowledge some limitations and suggest future research directions:</p><p><strong>Prosody Modeling</strong>: Although FastPitch improves pitch prediction, further work is needed to model other aspects of prosody, such as duration and energy contours, in a parallel and efficient manner.</p><p><strong>Multi-Speaker Adaptation</strong>: The FastPitch model presented in the paper is trained on a single-speaker dataset. Extending the model to handle multiple speakers and enable efficient speaker adaptation remains an open challenge.</p><p><strong>Deployment and Inference Optimization</strong>: While FastPitch offers faster inference times compared to autoregressive models, further optimization and deployment strategies could be explored to enable real-time applications and on-device inference.</p><p>Overall, FastPitch is a notable contribution to the field of text-to-speech synthesis, introducing parallel pitch prediction and demonstrating improved speech quality while maintaining computational efficiency. However, there is still room for further research and development to address the remaining limitations and expand the model’s capabilities.</p><h3>DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021</h3><p>DelightfulTTS is a text-to-speech (TTS) system developed by Microsoft for the Blizzard Challenge 2021, a prestigious annual evaluation of TTS systems. This system aims to generate high-quality and expressive speech while maintaining computational efficiency.</p><p><strong>Architecture</strong></p><p>DelightfulTTS is based on a non-autoregressive (NAR) architecture, which allows for parallel generation of mel-spectrograms, enabling faster inference compared to traditional autoregressive models. The acoustic model still faces the challenge of the one-to-many mapping problem, where multiple speech variations can correspond to the same input text. To address this issue, DelightfulTTS systematically models variation information from both explicit and implicit perspectives:</p><ol><li>For explicit factors like speaker ID and language ID, lookup embeddings are used during training and inference.</li><li>For explicit factors like pitch and duration, the values are extracted from paired text-speech data during training, and dedicated predictors are employed to estimate these values during inference.</li><li>For implicit factors like utterance-level and phoneme-level prosody, reference encoders are used to extract the corresponding values during training, while separate predictors are employed to predict these values during inference.</li></ol><p>The acoustic model in DelightfulTTS is built upon non-autoregressive generation models like FastSpeech, incorporating an improved Conformer module to better capture local and global dependencies in the mel-spectrogram representation. By explicitly modeling variation information from multiple perspectives and leveraging non-autoregressive architectures, DelightfulTTS aims to generate high-quality and expressive speech while addressing the one-to-many mapping challenge inherent in text-to-speech synthesis.</p><p><strong>Schema:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*d-FlBQsZuKE0BQXFiKfaaA.png" /></figure><p><strong>Conformer Architecture</strong></p><p>The Conformer architecture is a variant of the Transformer model that integrates both convolutional neural networks (CNNs) and self-attention mechanisms. It was originally proposed for end-to-end speech recognition tasks and later adopted for TTS systems due to its ability to capture both local and global dependencies in the input sequences. In the context of TTS, the Conformer architecture is typically used in the acoustic model, which is responsible for generating the mel-spectrogram representation of speech from the input text.</p><p>Mathematically, the Conformer block can be represented as follows: Let <strong>X</strong> be the input sequence. The key components of the Conformer block used in the acoustic model are:</p><ul><li><strong>Convolutional Feed-Forward Module</strong>: This module applies a 1D convolutional layer to the input sequence, capturing local dependencies.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*kMIDFEutRQXgNlg-6gy1QQ.png" /></figure><ul><li><strong>Depthwise Convolution Module</strong>: This module applies a depthwise 1D convolutional layer, further enhancing the modeling of local correlations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*aKKd1n06fbrqy29U6NTP-A.png" /></figure><ul><li><strong>Self-Attention Module</strong>: This module employs multi-headed self-attention, allowing the model to capture global dependencies and long-range interactions within the input sequence.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*aAgP3lwyailFIwa6U95nCw.png" /></figure><ul><li><strong>Second Convolutional Feed-Forward Module</strong>: Another convolutional feed-forward module is applied after the self-attention module, further processing the output.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*rjTtsDwPTtAS9g0-4IX1Ig.png" /></figure><p>Here, <strong>Conv1D</strong>, <strong>DepthwiseConv1D</strong>, and <strong>MultiHeadAttention</strong> represent the convolutional, depthwise convolutional, and multi-headed self-attention operations, respectively. The Conformer architecture combines the strengths of CNNs and self-attention, allowing the model to capture both local and global dependencies in the input sequence. This is particularly important for TTS tasks, where the model needs to generate mel-spectrograms of varying lengths while capturing the prosodic and acoustic characteristics of speech. In the DelightfulTTS model, the authors likely employed the Conformer architecture in the acoustic model to benefit from its ability to model both local and global dependencies, leading to improved prosody and audio fidelity in the synthesized speech.</p><p><strong>Variation Information Modeling</strong></p><p>Text-to-speech (TTS) is a typical one-to-many mapping problem where there could be multiple varying speech outputs (e.g., different pitch, duration, speaker, prosody, emotion, etc.) for a given text input. It is critical to model these variation information in speech to improve the expressiveness and fidelity of synthesized speech. While previous works have tried different methods to model the information, they focus on a specific aspect and cannot model in a comprehensive and systematic way. In this paper, considering that different variation information can be complementary to each other, the authors propose a unified way to model them in the proposed variance adaptor (as shown in Figure 1c).</p><p><strong>Categorization of Variation Information</strong></p><p>Observing that some variation information can be obtained implicitly (e.g., pitch can be extracted by some tools) or explicitly (e.g., utterance-level prosody can only be learned by the model), the authors categorize all the information they model as follows:</p><ol><li><strong>Explicit Modeling</strong>: Language ID, Speaker ID, Pitch, Duration</li><li><strong>Implicit Modeling</strong>: Utterance-level prosody, Phoneme-level prosody</li></ol><p>For speaker and language ID, lookup embeddings are used in training and inference. For pitch and duration, the values are extracted from paired text-speech data in training, and two predictors are used to predict the values in inference.</p><p>For utterance-level and phoneme-level prosody, two reference encoders are used to extract the values in training , and two separate predictors are used to predict the values in inference. The two reference encoders are both made up of convolution and RNN layers. Utterance-level prosody vector is obtained by the last RNN hidden and a style token layer. Phoneme-level prosody vectors are obtained by using the outputs of the phoneme encoder (phoneme-level) as a query to attend to the outputs of the mel-spectrogram reference encoder (frame-level). Different from, the authors do not use VAE but directly use the latent representation as the phoneme-level vector for training stability . The utterance-level prosody predictor contains a GRU layer followed by a bottleneck module to predict the prosody vector. The phoneme-level prosody predictor takes both the outputs of the text encoder and the utterance-level prosody vector as input. With the help of the utterance-level prosody vector, the authors do not need an autoregressive prosody predictor as in for faster inference.</p><p>By unifying explicit and implicit information in different granularities (language-level, speaker-level, utterance-level, phoneme-level) in the variance adaptor, the authors aim to achieve better expressiveness in prosody and controllability in pitch and duration.</p><p><strong>Key Contributions</strong></p><ul><li><strong>Non-Autoregressive Architecture</strong>: DelightfulTTS employs a non-autoregressive architecture, enabling faster inference compared to autoregressive models while maintaining high speech quality.</li><li><strong>Explicit Prosody Modeling</strong>: The system explicitly models pitch and energy contours, in addition to duration, to capture the prosodic variations in speech, leading to more natural and expressive synthesized speech.</li><li><strong>Multi-Speaker Capabilities</strong>: DelightfulTTS supports multi-speaker synthesis by conditioning the generation process on speaker embeddings, enabling efficient speaker adaptation.</li></ul><p><strong>Limitations and Future Work</strong></p><p>While DelightfulTTS represents a significant advancement in non-autoregressive TTS, the authors acknowledge some limitations and suggest future research directions:</p><ul><li><strong>Robustness to Long Inputs</strong>: The system’s performance may degrade for very long input sequences, and further work is needed to improve robustness and consistency for such cases.</li><li><strong>Fine-Grained Prosody Control</strong>: While DelightfulTTS models pitch, energy, and duration, additional work could explore more fine-grained control over prosodic aspects, such as emphasis and phrasing.</li><li><strong>Deployment and Optimization</strong>: Further optimization and deployment strategies could be explored to enable real-time applications and on-device inference, leveraging the non-autoregressive architecture’s potential for efficiency.</li></ul><p>Overall, DelightfulTTS is a notable contribution to the field of text-to-speech synthesis, demonstrating the potential of non-autoregressive architectures and explicit prosody modeling for generating high-quality and expressive speech. The system’s performance in the Blizzard Challenge 2021 highlights its competitiveness and the progress made in this domain.</p><h3>AR and non-AR TTS Models Review Summary</h3><p>Based on my research, I haven’t found a clear convergence between the areas of autoregressive (AR) and non-autoregressive (NAR) text-to-speech models in terms of combining or composing them into a single unified model. The key requirements for NAR models are sustainability, the ability to guarantee stability in the generated audio sequences, balanced quality, and fast generation speed, which is a significant advantage of the NAR architecture. Both the FastPitch and DelightfulTTS models meet these criteria. However, I prefer the DelightfulTTS model because it is more complex compared to FastPitch and can potentially provide more diverse results. The DelightfulTTS model’s complexity and diversity could be further enhanced by incorporating ideas from the StyledTTS2 model, which uses diffusion as an AR variator to make the speech more unique and expressive based on the context. While the diffusion-based approach in StyledTTS2 can introduce expressiveness, the NAR nature of DelightfulTTS can contribute to the stability of the generated speech. Therefore, a potential direction could be to explore a hybrid approach that combines the strengths of DelightfulTTS’s NAR architecture with the expressive capabilities of the diffusion-based variator from StyledTTS2. Such a hybrid model could leverage the stability and efficiency of the NAR framework while incorporating the contextual expressiveness and diversity offered by the diffusion-based variator. However, it’s important to note that integrating these two approaches into a single unified model may pose challenges, and further research is needed to investigate the feasibility and effectiveness of such a hybrid approach.</p><p>This brings us to the next article…</p><h3>Next chapter</h3><p><a href="https://medium.com/@peechapp/text-to-speech-models-part-6-delightfultts-implementation-and-training-ceb4c94fd5fb">Part 6: DelightfulTTS implementation and training 🛠️</a></p><p>Author: <a href="https://www.linkedin.com/in/nick-ovchinnikov/">Nick Ovchinnikov</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*TOD7iYtYNFfHgpIcqfPxlw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ce8de9c9e9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text-To-Speech Models. Part 4: Autoregressive and hybrid models review]]></title>
            <link>https://medium.com/@peechapp/text-to-speech-models-part-4-autoregressive-and-hybrid-models-review-abeb2b5ec3ec?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/abeb2b5ec3ec</guid>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[speech]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 09:15:57 GMT</pubDate>
            <atom:updated>2024-06-11T14:59:59.773Z</atom:updated>
            <content:encoded><![CDATA[<h3>Autoregressive and Hybrid Open-Source TTS Models Review</h3><h3>Read the Full Series of ML TTS exploration</h3><ul><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-1-intro-little-theory-and-math-0ffa5d3e0e3f">Part 1: Intro, Little Theory and Math 📘</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-2-autoregressive-models-world-636d8aa0932d">Part 2: Autoregressive models world 🌍</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-3-non-autoregressive-models-hideout-f45e46a4f4bd">Part 3: Non-autoregressive models hideout 🕵️‍♂️</a></li><li>👉Part 4: Autoregressive and hybrid models review 📊</li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-5-non-autoregressive-models-review-7ce8de9c9e9d">Part 5: Non-autoregressive models review📝</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-6-delightfultts-implementation-and-training-ceb4c94fd5fb">Part 6: DelightfulTTS implementation and training 🛠️</a></li></ul><h3><strong>AR and hybrid models</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*QbBnzpvTKs1YfbmuSQUhIw.jpeg" /><figcaption>Typical hybrid models</figcaption></figure><p>In the following sections, I will delve into the realm of autoregressive (AR) text-to-speech models, renowned for their ability to capture intricate sequential dependencies and generate expressive, natural-sounding speech. While exploring their architectures, training methodologies, and key innovations. Despite the challenges, AR models have been at the forefront of speech synthesis research.</p><h3><strong>SunoAI Bark</strong></h3><p>My research has begun with the discovery of an incredible open-source solution that can produce incredible results: <a href="https://github.com/suno-ai/bark">SunoAI Bark</a></p><p>Bark is obviously AR model, but how it works? I found this great repo: <a href="https://github.com/gitmylo/audio-webui">audio-webui</a> and author tried to found out <a href="https://github.com/gitmylo/audio-webui/wiki/how-bark-works#implementations--in-chronological-order-">how bark works</a>, but:</p><blockquote><strong>📓 Note</strong></blockquote><blockquote>I’m still unsure how bark works. I am trying to figure out how it works though. My current knowledge goes here.</blockquote><p>The author is researching voice cloning using the SunoAI Bark model and identified three different models used in the generation process: Semantics, Coarse, and Fine. The author notes that the current method of generating semantic tokens is problematic, as it generates tokens based on the input text rather than the actual audio.</p><p>The author proposes three potential way for creation of speaker files for voice cloning:</p><ol><li>Find out how Bark generates semantic tokens (may not be publicly available).</li><li>Train a neural network to convert audio files to semantic tokens (time-consuming, but effective).</li><li>Use a speech converter to change the input audio without editing semantics (not perfect, but better than current methods).</li></ol><p>The author has implemented Method 3 using coqui-ai/TTS and achieved decent voice cloning results, but with limitations.</p><blockquote><strong>Results</strong>: Decent voice cloning, not near perfect though. Better than previous methods, but struggles with some voices and accents. These issues lie in the transfer step.</blockquote><p>They also explored Method 1, but were unable to find a suitable model.</p><blockquote><strong>Pre:</strong> It looks like bark uses <a href="https://github.com/lucidrains/audiolm-pytorch">AudioLM</a> for the semantic tokens! I’m not sure if they use a different model though. I’ll have to test that. But if they don’t use a pre-trained model. I can do step 2.</blockquote><blockquote><strong>Results</strong>: No, it doesn’t look like i can find a model, i did succeed in creating same-size vector embeddings, but the vectors use the wrong tokens.</blockquote><p>Method 2, which involves creating a quantizer based on data, has shown promising results, achieving convincing voice cloning with some limitations.</p><blockquote><strong>Pre</strong>: What if instead of training a whole model, i only create a quantizer based on a bunch of data?</blockquote><blockquote><strong>Results</strong>: Successful voice cloning, can be very convincing. Still some limitations. But better than anything I’ve seen done with bark before.</blockquote><p>I can’t tell you more, please, share any details if you find better explanations of the Bark. AR nature of the model is obvious, that’s for sure. I faced with the limitations from the first touch — model can’t generate audio more than 13–14 sec length.</p><p>Several issues about this problem are here:</p><ul><li><a href="https://github.com/suno-ai/bark/issues/79">Limited to 13 seconds?</a></li><li><a href="https://github.com/suno-ai/bark/issues/21">Arbitrarily long text</a></li></ul><p>Comment about it from issue <a href="https://github.com/suno-ai/bark/issues/36">#36 Inquiry on Maximum input character text prompt Length</a>:</p><blockquote>right now the output is limited by the context window of the model (1024) which equates to about 14s. So text should be around that duration (meaning ~2–3 sentences). For longer texts you can either do them one at a time (and use the same history prompt to continue the same voice) or feed the first generation as the history for the second. i know that that’s still a bit inconvenient. will try to add better support for that in the next couple of days</blockquote><p>You can implement the following <a href="https://github.com/suno-ai/bark/pull/84#issuecomment-1523433448">approach</a>, split sentences and synthesize the audio chunks:</p><pre>from bark import generate_audio,preload_models<br>from scipy.io.wavfile import write as write_wav<br>import numpy as np<br>import nltk<br><br>nltk.download(&#39;punkt&#39;)<br>preload_models()<br><br>long_string = &quot;&quot;&quot;<br>Bark is a transformer-based text-to-audio model created by [Suno](https://suno.ai/). Bark can generate highly realistic, multilingual speech as well as other audio - including music, background noise and simple sound effects. The model can also produce nonverbal communications like laughing, sighing and crying. To support the research community, we are providing access to pretrained model checkpoints ready for inference.<br>&quot;&quot;&quot;<br><br>sentences = nltk.sent_tokenize(long_string)<br><br># Set up sample rate<br>SAMPLE_RATE = 22050<br>HISTORY_PROMPT = &quot;en_speaker_6&quot;<br><br>chunks = [&#39;&#39;]<br>token_counter = 0<br><br>for sentence in sentences:<br> current_tokens = len(nltk.Text(sentence))<br> if token_counter + current_tokens &lt;= 250:<br>  token_counter = token_counter + current_tokens<br>  chunks[-1] = chunks[-1] + &quot; &quot; + sentence<br> else:<br>  chunks.append(sentence)<br>  token_counter = current_tokens<br><br># Generate audio for each prompt<br>audio_arrays = []<br>for prompt in chunks:<br>    audio_array = generate_audio(prompt,history_prompt=HISTORY_PROMPT)<br>    audio_arrays.append(audio_array)<br><br># Combine the audio files<br>combined_audio = np.concatenate(audio_arrays)<br><br># Write the combined audio to a file<br>write_wav(&quot;combined_audio.wav&quot;, SAMPLE_RATE, combined_audio)</pre><p>Also you can use batching:</p><pre>from IPython.display import Audio<br><br>from transformers import AutoProcessor<br>from transformers import BarkModel<br><br>model = BarkModel.from_pretrained(&quot;suno/bark-small&quot;)<br>processor = AutoProcessor.from_pretrained(&quot;suno/bark-small&quot;)<br><br>text_prompt = [<br>    &quot;Let&#39;s try generating speech, with Bark, a text-to-speech model&quot;,<br>    &quot;Wow, batching is so great!&quot;,<br>    &quot;I love Hugging Face, it&#39;s so cool.&quot;<br>]<br><br>inputs = processor(text_prompt).to(device)<br><br>with torch.inference_mode():<br>  # samples are generated all at once<br>  speech_output = model.generate(**inputs, do_sample = True, fine_temperature = 0.4, coarse_temperature = 0.8)<br><br># let&#39;s listen to the output samples!<br><br>Audio(speech_output[2].cpu().numpy(), rate=sampling_rate)</pre><p>But the butch generation creates a new problem. The quality of the audio is dramatically reduced, and it generates strange noises, much more often than in the case of generation in order. I found a great description in the comment: <a href="https://github.com/suno-ai/bark/issues/21#issuecomment-1516990620">output quality seems worse if you pack it like that</a></p><p>One more problem that I found is empty sounds if your text chunks are different sizes. Sometimes it’s a random noise like background voices etc. So, you need to split the text carefully, it’s very important.</p><p>Bark occasionally hallucinates, which can lead to unexpected and inaccurate results. Unfortunately, it is impossible to detect this issue during the generation process, making it challenging to identify and correct. Furthermore, even during the review process, hallucinations can be difficult to spot, as they may be subtle or blend in with the rest of the generated content. This can result in incorrect or misleading information being presented as factual, which can have serious consequences in certain applications.</p><p><strong>SunoAI summary</strong></p><p>From what I understand, the issue of long context is not a priority for the Bark team and is largely being overlooked in the issues discussion. The only solution they’ve proposed is to use batches, but this approach has its limitations and doesn’t work well. Moreover, due to the text context, it can’t be processed accurately. Specifically, a small context of 2–3 sentences tends to disregard what happened in the previous passage, which can significantly impact the intonation and tempo of a speaker. Bark occasionally hallucinates, which can lead to unexpected and inaccurate results.</p><h3><strong>VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers</strong></h3><p>Samples on the <a href="https://www.microsoft.com/en-us/research/project/vall-e-x/">demo page</a>.</p><p>VALL-E is a novel language model approach for text-to-speech (TTS) synthesis that leverages large-scale, diverse, and multi-speaker speech data. Unlike traditional cascaded TTS systems, VALL-E treats TTS as a conditional language modeling task, using discrete audio codec codes as intermediate representations. This approach enables VALL-E to synthesize high-quality, personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt.</p><p><strong>Key Components:</strong></p><ol><li><strong>Neural Codec Model</strong>: VALL-E uses a pre-trained neural audio codec model, <a href="https://github.com/facebookresearch/encodec">EnCodec</a>, to convert audio waveforms into discrete acoustic codes. This model is trained on a large dataset of speech and can reconstruct high-quality waveforms even for unseen speakers.</li><li><strong>Language Model</strong>: VALL-E is trained as a language model, using the discrete acoustic codes as input and output. This allows the model to learn the patterns and structures of speech, enabling it to generate coherent and natural-sounding speech.</li><li><strong>In-Context Learning</strong>: VALL-E has strong in-context learning capabilities, similar to GPT-3, which enables it to adapt to new speakers and acoustic environments with minimal additional training data.</li></ol><p><strong>The Problem Formulation is valid for Bark as well (except dimensions):</strong></p><p>Given a dataset <strong>D = {xᵢ, yᵢ} </strong>where <strong>y</strong> is an audio sample and:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*Eb_VC8ApB0n1l_Ql-DMpFA.png" /></figure><p>is its corresponding phoneme transcription, we use a pre-trained neural codec model to encode each audio sample into discrete acoustic codes, denoted as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*6-Hf3LXVmb1y855xtpJRbg.png" /></figure><p>where <strong>C</strong> represents the two-dimensional acoustic code matrix, and <strong>T</strong> is the downsampled utterance length. The row vector of each acoustic code matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/45/1*5MwW1fRM4FzO0Bt-mduLkw.png" /></figure><p>represents the eight codes for frame <strong>t </strong>and the column vector of each acoustic code matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/1*KS1PfQAN2gXcdazGz1QDkw.png" /></figure><p>represents the code sequence from the <strong>j-th</strong> codebook, where <strong>j ∈ {1, …, 8}</strong>. After quantization, the neural codec decoder is able to reconstruct the waveform, denoted as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*uDXvpyAJIXaNtboOU2Vu4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*6bBpppjts2EspRWR-9apEw.png" /><figcaption>Vall-E schema</figcaption></figure><p><strong>Based on the paper, it appears that the proposed model combines the principles of Autoregressive (AR) and Non-Linear Autoregressive (NAR) models:</strong></p><blockquote>The combination of the AR model and the NAR model provides a good trade-off between speech quality and inference speed. On the one hand, the rate of the generated speech should be consistent with the enrolled recording, and it is hard to train a length predictor for different speakers since their speaking speed may be very diverse. In this case, the AR model is a more natural choice with its flexibility for acoustic sequence length prediction. On the other hand, for the consecutive stages, as the number of output slots follows the sequence length of the first stage, NAR can reduce the time complexity from <strong>O(T)</strong> to <strong>O(1)</strong>. Overall, the prediction of <strong>C</strong> can be modeled as:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8V0KoGfnU1NsJCmHYw9rxA.png" /></figure><p><strong>VALL-E implementations</strong></p><p>Note that there is currently no official implementation of VALL-E available. Even ETA for the official release is unknown <a href="https://github.com/microsoft/unilm/issues/977">VALL-E ETA?</a> The <a href="https://github.com/microsoft/unilm/tree/master/valle">official repository</a> currently contains an empty space, awaiting further updates. However, for those interested in experimenting with VALL-E, an unofficial PyTorch implementation is available at <a href="https://github.com/lifeiteng/vall-e">unofficial PyTorch implementation of VALL-E</a>. This implementation can be trained on a single GPU and is widely used in the community.</p><blockquote><a href="https://github.com/lifeiteng/vall-e/issues/53">After eight hours of training, the model can synthesize natural speech!</a></blockquote><blockquote>code branch `position` <a href="https://github.com/lifeiteng/vall-e/pull/46">PR #46</a></blockquote><blockquote>egs <a href="https://github.com/lifeiteng/vall-e/tree/position/egs/ljspeech">https://github.com/lifeiteng/vall-e/tree/position/egs/ljspeech</a></blockquote><blockquote>synthesised speech <a href="https://github.com/lifeiteng/vall-e/tree/position/egs/ljspeech/demos">https://github.com/lifeiteng/vall-e/tree/position/egs/ljspeech/demos</a></blockquote><blockquote>The next step is to verify `in-context learning capabilities` of VALL-E on a large dataset.</blockquote><p>And I found one more training story: <a href="https://github.com/lifeiteng/vall-e/issues/58">After 100 epochs training, the model can synthesize natural speech on LibriTTS</a></p><blockquote>I trained vall-e on LibriTTS about 100 epochs (took almost 4 days on 8 A100 GPUs) and I obtained plausible synthesized audio.</blockquote><blockquote>Here is a demo.</blockquote><blockquote>[1] prompt : <a href="https://drive.google.com/file/d/149pHqb6TZzVwhF1vRN50H8A4AEYShpfp/view?usp=share_link">prompt_link</a> synthesized audio: <a href="https://drive.google.com/file/d/189MpDf7f3nL-RfO3MtgtQ0APuwRygA0M/view?usp=share_link">synt_link</a></blockquote><blockquote>[2] prompt : <a href="https://drive.google.com/file/d/149pHqb6TZzVwhF1vRN50H8A4AEYShpfp/view?usp=share_link">prompt_link</a> ground truth : <a href="https://drive.google.com/file/d/1rfsQD11Dtp7bzng73qGTU7H_YKSdeOdZ/view?usp=share_link">gt_link</a> synthesized audio: <a href="https://drive.google.com/file/d/1Y4llFLT3Z8proFU_HeasRYGj656pLtcm/view?usp=share_link">synt_link</a></blockquote><blockquote>[3] prompt : <a href="https://drive.google.com/file/d/1N9EqA1GWJOBIZbjqpHidrEJs6sGtVuyu/view?usp=share_link">prompt_link</a> synthesized audio : <a href="https://drive.google.com/file/d/1-W8QnhliE5UcFx8fn8B20KRlls5LGfiT/view?usp=share_link">synt_link</a></blockquote><blockquote>[4] prompt : <a href="https://drive.google.com/file/d/1N9EqA1GWJOBIZbjqpHidrEJs6sGtVuyu/view?usp=share_link">prompt_link</a> ground truth: <a href="https://drive.google.com/file/d/1EIz686OEQ4-zep_JSN_WKriSIFiwz-Xc/view?usp=share_link">gt_link</a> synthesized audio : <a href="https://drive.google.com/file/d/1z9EUcQdSf4QD_8KH3mcZpAdRMZZW9XpH/view?usp=share_link">synt_link</a></blockquote><blockquote>The model I trained has worse quality than the original vall-e because of the dataset amount. However, it has promising quality in clean audio. I’m not sure whether I can share my pre-trained LibriTTS model. If I can, I would like to share the pre-trained LibriTTS model.</blockquote><p>Model weights:</p><blockquote>Sorry for late reply. This is the model that I trained.</blockquote><blockquote>google drive link: <a href="https://drive.google.com/file/d/1pKvS56NnzVCYqhbjoFqL_JQ8i32UixbL/view?usp=sharing">https://drive.google.com/file/d/1pKvS56NnzVCYqhbjoFqL_JQ8i32UixbL/view?usp=sharing</a></blockquote><p>One more unofficial PyTorch implementation of <a href="https://github.com/enhuiz/vall-e">VALL-E</a>, based on the <a href="https://github.com/facebookresearch/encodec">EnCodec</a> tokenizer.</p><p><strong>VALL-E summary</strong></p><p>So, VALL-E is an AR or NAR model? VALL-E is a hybrid model that leverages the strengths of both AR and NAR architectures.</p><p>In section <em>6 Conclusion, Limitations, and Future Work</em> you can see one important note:</p><blockquote><strong>Synthesis robustness:</strong> We observe that some words may be unclear, missed, or duplicated in speech synthesis. It is mainly because the phoneme-to-acoustic language part is an autoregressive model, in which disordered attention alignments exist and no constraints to solving the issue. The phenomenon is also observed in vanilla Transformer-based TTS, which was addressed by applying non-autoregressive models or modifying the attention mechanism in modeling. In the future, we would like to leverage these techniques to solve the issue.</blockquote><p>It’s interesting to note that the authors acknowledge this limitation and plan to explore techniques from non-autoregressive models or modified attention mechanisms to address this issue in future work. Even state-of-the-art AR models struggle with limitations like disordered attention alignments, leading to synthesis robustness issues.</p><h3><strong>StyledTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models</strong></h3><p><a href="https://github.com/yl4579/StyleTTS2">Github repo</a></p><p>StyleTTS 2 is a state-of-the-art text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis.</p><p>By modeling styles as a latent random variable through diffusion models, StyleTTS 2 is able to generate the most suitable style for the input text in diverse speech synthesis. The use of large pre-trained SLMs, such as WavLM, as discriminators with differentiable duration modeling enables end-to-end training and improves speech naturalness.</p><h4><strong>Style Diffusion</strong></h4><p><strong>StyleTTS 2 </strong>models speech <strong>x</strong> as a conditional distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*yKnIkawuTpIRutjt73loNg.png" /></figure><p>where <strong>t</strong> represents the phonetic content, and <strong>s</strong> is a latent variable representing the generalized speech style. This style variable <strong>s</strong> is sampled using Efficient Diffusion Models (EDM), which follow the combined probability flow and time-varying Langevin dynamics. By modeling styles as a latent random variable through diffusion models, StyleTTS 2 can generate the most suitable style for the input text, enabling diverse speech synthesis.</p><p><strong>Adversarial Training with Large SLMs</strong></p><p>StyleTTS 2 employs large pre-trained SLMs, such as WavLM, as discriminators during training. This adversarial training, combined with the proposed differentiable duration modeling, results in improved speech naturalness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*XNzZMhEtCE3R0fLXM67NXg.png" /></figure><p><strong>Key Contributions</strong></p><ol><li><strong>Differentiable Upsampling and Fast Style Diffusion</strong>: StyleTTS 2 introduces a differentiable upsampling method that allows for generating speech samples during training in a fully differentiable manner, similar to inference. These generated samples are used to optimize the loss function involving the large pre-trained SLM, enabling end-to-end training of all components.</li><li><strong>Efficient Style Sampling</strong>: To enable fast style sampling during inference, StyleTTS 2 uses the ancestral DPM-2 solver instead of the 2nd-order Heun method, allowing for fast and diverse sampling without significantly impacting inference speed.</li><li><strong>Multispeaker Modeling</strong>: For multispeaker settings, StyleTTS 2 models <strong>p(s|t, c)</strong>, where <strong>c</strong> is a speaker embedding obtained from a reference audio of the target speaker. This speaker embedding is injected into the transformer model using adaptive layer normalization.</li></ol><p><strong>Known Issues</strong></p><p>It’s worth noting that StyleTTS 2 has a known issue where loud white noise can occur on short texts when the `embedding_scale` parameter is set to a value greater than 1 (see <a href="https://github.com/yl4579/StyleTTS2/issues/46">GitHub issue #46</a>).</p><p><strong>Hybrid AR-NAR Approach perspective</strong></p><p>StyleTTS 2 presents a novel and promising approach to text-to-speech synthesis by combining the strengths of both autoregressive (AR) and non-autoregressive (NAR) models. This hybrid architecture leverages the power of large pre-trained SLMs, which are typically autoregressive models, while incorporating the diversity and expressiveness offered by diffusion models, which are non-autoregressive. By combining the pre-trained SLM with the style diffusion process, StyleTTS 2 effectively creates a hybrid architecture that combines the strengths of both AR and NAR models. The SLM provides a strong foundation for capturing sequential dependencies and generating high-quality speech, while the diffusion process introduces diversity and expressiveness, enabling the generation of natural-sounding speech with appropriate styles and characteristics.</p><p>The combination of style diffusion and adversarial training with large SLMs has proven to be a powerful approach for generating natural and expressive speech.</p><h3>Next chapter</h3><p><a href="https://medium.com/@peechapp/text-to-speech-models-part-5-non-autoregressive-models-review-7ce8de9c9e9d">Part 5: Non-autoregressive models review📝</a></p><p>Author: <a href="https://www.linkedin.com/in/nick-ovchinnikov/">Nick Ovchinnikov</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*jejLQnvaFzcI_UDXvn9nUw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abeb2b5ec3ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text-To-Speech Models. Part 3: Non-Autoregressive models hideout]]></title>
            <link>https://medium.com/@peechapp/text-to-speech-models-part-3-non-autoregressive-models-hideout-f45e46a4f4bd?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f45e46a4f4bd</guid>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[speech]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 09:03:23 GMT</pubDate>
            <atom:updated>2024-06-11T15:00:28.553Z</atom:updated>
            <content:encoded><![CDATA[<h3>Non-Autoregressive Models Hideout</h3><h3>Read the Full Series of ML TTS exploration</h3><ul><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-1-intro-little-theory-and-math-0ffa5d3e0e3f">Part 1: Intro, Little Theory and Math 📘</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-2-autoregressive-models-world-636d8aa0932d">Part 2: Autoregressive models world 🌍</a></li><li>👉Part 3: Non-autoregressive models hideout 🕵️‍♂️</li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-4-autoregressive-and-hybrid-models-review-abeb2b5ec3ec">Part 4: Autoregressive and hybrid models review 📊</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-5-non-autoregressive-models-review-7ce8de9c9e9d">Part 5: Non-autoregressive models review📝</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-6-delightfultts-implementation-and-training-ceb4c94fd5fb">Part 6: DelightfulTTS implementation and training 🛠️</a></li></ul><h3><strong>Non-autoregressive models (NAR)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*eNkCbqIomLIzDL0GN3Z1og.jpeg" /></figure><p>Non-autoregressive (NAR) sequence generation is an alternative approach to autoregressive (AR) models, where the entire output sequence is generated in parallel, without relying on previously generated tokens. Unlike AR models, which generate the sequence one token at a time in an autoregressive manner, NAR models do not have any sequential dependencies during the generation process. The key advantage of NAR models is their computational efficiency and speed. By generating the entire sequence in a single pass, NAR models can significantly reduce the inference time compared to AR models, which can be particularly beneficial for applications that require real-time or low-latency performance.</p><h3><strong>Time Complexity Analysis</strong></h3><p>The time complexity of NAR models is constant, denoted as <strong>O(1)</strong>, as the generation of the entire sequence is performed in a single pass, regardless of the sequence length. This is in contrast to AR models, which have a linear time complexity or <strong>O(n)</strong>.</p><h4><strong>Definition</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e76b0f36c0fa59f7faa2a50d0edf6ce9/href">https://medium.com/media/e76b0f36c0fa59f7faa2a50d0edf6ce9/href</a></iframe><p>Note that in a non-autoregressive model, the probability of each token is independent of the previous tokens, in contrast to AR models, which model the conditional probabilities using the chain rule.</p><h3><strong>NAR model schema</strong></h3><p><em>Note: over-oversimplified schema, shouldn’t be considered seriously at all. Shows that every token is independent.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*WoF_w9uVQRTpWa0Pdehl0Q.png" /><figcaption>Dummy NAR schema</figcaption></figure><h3><strong>Cons of NAR models</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*mhbQgre_LZ_sB-PrDwkwsQ.png" /></figure><p>NAR models generate each token independently, without considering the context of the previous tokens, models do not capture sequential dependencies between tokens, which can result in generated sequences that do not follow the natural flow of language. <strong>The independence assumption in the output space ignores the real dependency between target tokens</strong>. NAR models are not well-suited to capture hierarchical structures in language, such as syntax and semantics.</p><p>Based on the nature NAR models may generate repetitive phonemes or sounds, leading to a lack of naturalness and coherence in the synthesized speech, models may generate shorter or broken utterances, failing to capture the full meaning or context of the input text. NAR models may generate speech that sounds unnatural or robotic, indicating a lack of understanding of the acoustic properties of speech, and also may struggle to capture long-range dependencies between tokens, leading to a lack of coherence and fluency in the generated sequence.</p><h4><strong>Text-to-Speech Non-Autoregressive models</strong></h4><p>Despite their limitations, Non-Autoregressive (NAR) models are still widely used in Text-to-Speech (TTS) systems. One of the main advantages of NAR models is their ability to generate speech much faster than Autoregressive (AR) models, thanks to their reduced computational requirements. This makes them well-suited for real-time TTS applications where speed is crucial. While NAR models may not produce speech that is as natural-sounding as AR models, they can still generate high-quality speech that is sufficient for many use cases.</p><h4><strong>Side-by-side AR / NAR</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*dUVbPf3b_9RHueNzZyZrVA.png" /></figure><p>To enhance the quality of the synthesized audio in non-autoregressive (NAR) text-to-speech models, researchers have explored the idea of incorporating additional modules or blocks to predict various speech parameters. By explicitly modeling speech parameters, such as pitch, energy, and other prosodic features, the models can generate more expressive and natural-sounding speech.</p><p>For instance, in the FastPitch model, a dedicated pitch predictor module is introduced to generate pitch contours in parallel, which are then used to condition the mel-spectrogram generation.</p><p><strong>Schema of FastSpeech:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*sykPJ80j1OO7lLCE-4YAgQ.png" /></figure><p>While introducing additional modules and conditioning the generation process on various speech parameters can improve the quality of the synthesized audio, it is important to strike a balance between model complexity and computational efficiency. The non-autoregressive nature of these models is designed to achieve faster inference speeds compared to autoregressive models, and adding too many complex modules or dependencies could potentially negate this advantage.</p><h3>Next chapter</h3><p><a href="https://medium.com/@peechapp/text-to-speech-models-part-4-autoregressive-and-hybrid-models-review-abeb2b5ec3ec">Part 4: Autoregressive and hybrid models review 📊</a></p><p>Author: <a href="https://www.linkedin.com/in/nick-ovchinnikov/">Nick Ovchinnikov</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*EZpnduipHwh0HscvhbaxqQ.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f45e46a4f4bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text-To-Speech Models. Part 2: Autoregressive models world]]></title>
            <link>https://medium.com/@peechapp/text-to-speech-models-part-2-autoregressive-models-world-636d8aa0932d?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/636d8aa0932d</guid>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[autoregressive-model]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 08:57:37 GMT</pubDate>
            <atom:updated>2024-06-11T14:58:03.961Z</atom:updated>
            <content:encoded><![CDATA[<h3>Autoregressive M<strong>odels</strong> World</h3><h3>Read the Full Series of ML TTS exploration</h3><ul><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-1-intro-little-theory-and-math-0ffa5d3e0e3f">Part 1: Intro, Little Theory and Math 📘</a></li><li>👉Part 2: Autoregressive models world 🌍</li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-3-non-autoregressive-models-hideout-f45e46a4f4bd">Part 3: Non-autoregressive models hideout 🕵️‍♂️</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-4-autoregressive-and-hybrid-models-review-abeb2b5ec3ec">Part 4: Autoregressive and hybrid models review 📊</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-5-non-autoregressive-models-review-7ce8de9c9e9d">Part 5: Non-autoregressive models review📝</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-6-delightfultts-implementation-and-training-ceb4c94fd5fb">Part 6: DelightfulTTS implementation and training 🛠️</a></li></ul><p>Any structured data can be transformed into a sequence. For instance, speech can be represented as a sequence of waveforms. Neural sequence generation methods can be used to generate such sequences, and there are two main categories of models for this purpose: <strong>autoregressive (AR)</strong> and <strong>non-autoregressive (non-AR)</strong> sequence generation methods.</p><p><em>I believe, that any sequence can be generated using neural sequence generation methods (but not sure).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*kCdiaHtq4XzLUa8P_liNbg.jpeg" /></figure><h3><strong>Autoregressive models</strong></h3><p>Autoregressive (AR) sequence generation involves generating a sequence one token at a time in an autoregressive manner. In an AR model, the current value in the sequence is predicted based on the previous values and an error term (often referred to as <a href="https://en.wikipedia.org/wiki/White_noise">white noise</a>, which represents the unpredictable or random component).</p><p><strong>The definition from wikipedia: </strong><a href="https://en.wikipedia.org/wiki/Autoregressive_model"><strong>autoregressive model</strong></a><strong>:</strong></p><blockquote>In statistics, econometrics, and signal processing, an <strong>autoregressive</strong> (<strong>AR</strong>) <strong>model</strong> is a representation of a type of random process; as such, it is used to describe certain time-varying processes in nature, economics, behavior, etc. The autoregressive model specifies that the output variable depends linearly on its own previous values and on a <a href="https://en.wikipedia.org/wiki/Stochastic%20&quot;Stochastic&quot;">stochastic</a> term (an imperfectly predictable term); thus the model is in the form of a stochastic difference equation (or recurrence relation) which should not be confused with a <a href="https://en.wikipedia.org/wiki/Differential_equation%20&quot;Differential%20equation&quot;">differential equation</a>.</blockquote><p>The AR model’s ability to capture sequential dependencies allows it to generate speech with natural-sounding variations, for example — making it expressive. This is because the model can learn to predict the next value in the sequence based on the context provided by the previous values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1727903335017880223aa69d1824f99/href">https://medium.com/media/c1727903335017880223aa69d1824f99/href</a></iframe><h3><strong>Probabilistic Perspective</strong></h3><p>To define AR Sequence Generation in a probabilistic manner, we can use the chain rule to decompose the joint probability of the sequence into a product of conditional probabilities.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5296b2065b25bbb2baaa2a7d8d9bfe5/href">https://medium.com/media/f5296b2065b25bbb2baaa2a7d8d9bfe5/href</a></iframe><h3><strong>Inference</strong></h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6eb12a9d65a4da88b5ecc705a7c99b4/href">https://medium.com/media/d6eb12a9d65a4da88b5ecc705a7c99b4/href</a></iframe><h3><strong>AR model schema</strong></h3><p>Note: oversimplified schema, shouldn’t be considered seriously. SOS — start of the sequence, EOS — end of the sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*PLsOLk_kmYwW59UeYhegqA.png" /><figcaption>AR Schema</figcaption></figure><h4><strong>The best example of the AR world: Transformer &amp; LLM</strong></h4><p>Transformers, which are the backbone of many state-of-the-art natural language processing (NLP) and speech models, including Large Language Models (LLMs), use a mechanism called <strong>self-attention</strong> to model relationships between all pairs of tokens in a sequence. Self-attention allows the model to capture long-range dependencies and weigh the importance of each token relative to others, enabling it to focus on the most relevant tokens when generating the next token.</p><p>The attention mechanism in Transformers can be thought of as a weighted sum of the input tokens, where the weights represent the importance or relevance of each token for the current prediction. This allows the model to selectively focus on the most relevant tokens when generating the output.</p><p>Mathematically, the attention mechanism can be represented as <strong>QKV attention</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*iCWdPRLLiIK9UdMkilNjBw.png" /><figcaption>QKV attention</figcaption></figure><p>where:</p><ul><li><strong>Q</strong> (Query) represents the current token or context for which we want to find relevant information.</li><li><strong>K</strong> (Key) and <strong>V</strong> (Value) represent the input tokens and their corresponding values or representations.</li><li><strong>dₖ</strong> is the dimensionality of the key vectors, and the scaling factor <strong>√dₖ</strong> is used for numerical stability.</li><li>The softmax function is applied to the scaled dot-product of the query and key vectors, resulting in a set of weights that sum to 1. These weights determine the relative importance of each input token for the current prediction.</li><li>The weighted sum of the value vectors, <strong>V</strong>, is then computed using the attention weights, producing the final output of the attention mechanism.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*j16K5q2ry__UXJLUyS_nrQ.png" /><figcaption>QKV attention</figcaption></figure><p>The self-attention mechanism in Transformers allows the model to capture dependencies between tokens in a sequence, regardless of their position or distance. This has made Transformers highly effective for various natural language processing tasks, including language modeling, machine translation, and text generation.</p><h3><strong>Cons of Autoregressive (AR) models</strong></h3><p>While the autoregressive nature of AR models allows them to capture sequential dependencies and generate expressive sequences, it also introduces a significant drawback: slow inference time, especially for long sequences.</p><h3><strong>Time Complexity Analysis</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*FD_7uDsQ2Xh30UwD3Y2OEQ.png" /></figure><p>The time complexity of an algorithm or model refers to how the computational time required for execution scales with the input size. In the case of AR models, the input size is the sequence length, denoted as <strong>n</strong>.</p><p>During inference, AR models generate sequences one token at a time, using the previously generated tokens to predict the next token. This sequential generation process means that the model needs to process each token in the sequence, resulting in a time complexity of <strong>O(n)</strong>. In other words, the inference time grows linearly with the sequence length.</p><p>This linear time complexity can become a significant bottleneck in real-time scenarios or when dealing with long sequences, such as generating long texts, speech utterances, or high-resolution images. Even with modern hardware and parallelization techniques, the sequential nature of AR models can make them computationally expensive and slow for certain applications.</p><p>The slow inference time of AR models can be particularly problematic in the following scenarios:</p><ol><li><strong>Real-time speech synthesis</strong>: In applications like virtual assistants or text-to-speech systems, the ability to generate speech in real-time is crucial for a seamless user experience. AR models may struggle to keep up with the required speed, leading to noticeable delays or latency.</li><li><strong>Long-form text generation</strong>: When generating long texts, such as articles, stories, or reports, the inference time of AR models can become prohibitively slow, making them impractical for certain use cases.</li></ol><p>While AR models excel at capturing sequential dependencies and generating expressive sequences, their slow inference time can be a significant limitation in certain applications, especially those requiring real-time or low-latency performance or involving long sequences.</p><p>One of the inherent challenges in autoregressive (AR) models is the unavoidable loss of quality for longer sequences. This issue is visually represented in the following plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*1kDMoIV8pxL3pUYidA_Xdw.png" /><figcaption>The dispersion of predicted values increases</figcaption></figure><p>As you can see, the dispersion or spread of the predicted values increases as the sequence length grows. This phenomenon occurs by design in AR models, and it’s crucial to understand the underlying reasons behind it.</p><h3><strong>Harmful Biases in Autoregressive (AR) Generation</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*1mlHWqexIlaMkKlTsmfTaw.jpeg" /></figure><p>Autoregressive (AR) models are prone to several harmful biases that can negatively impact their performance and the quality of the generated sequences. These biases can arise from various factors, including the training process, the model architecture, and the nature of the data itself. Let’s explore the reasons behind these, why do we have such kind of troubles?</p><h4><strong>Error Propagation</strong></h4><p>Error propagation occurs when errors in the predicted tokens accumulate and affect the subsequent tokens, leading to a compounding effect, where the errors in the early tokens propagate to the later tokens. This can cause the generated sequence to deviate significantly from the true sequence, especially for longer sequences.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ffb8d632fa3a77ae71a418e0dc588df3/href">https://medium.com/media/ffb8d632fa3a77ae71a418e0dc588df3/href</a></iframe><p>The second term, represents the accumulated error propagation term, which grows as the sequence length increases.</p><h4><strong>Error-propagation term:</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*_h1nWgw5ZPm0d15g6NmN6g.png" /></figure><p><strong>The worst-case scenario occurs when the error happens on the first token in the sequence, x̂₁.</strong> In this case, the error propagates and accumulates through the entire sequence, leading to the maximum deviation from the true sequence.</p><p>In the context of text-to-speech synthesis, error propagation can lead to the generation of random noise or artifacts in the synthesized audio, making it difficult to avoid and potentially resulting in an unnatural or distorted output.</p><h4><strong>Label Bias</strong></h4><p>Label bias occurs when the normalization constraint over the vocabulary items at each decoding step in AR models leads to learning miscalibrated distributions over tokens and sequences. This can happen because the model is forced to assign probabilities that sum to 1 over the entire vocabulary, even if some tokens or sequences are highly unlikely or impossible in the given context.</p><p>Mathematically, the probability distribution over the vocabulary items at time step <strong>t</strong> is typically computed using a softmax function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qYXL5Hi4EVr8obrprDMqUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*KF7pvVeYCgdIjxsisXiE_w.png" /></figure><p><strong>W</strong> is the weight matrix, and <strong>b</strong> is the bias term.</p><p>This normalization constraint can lead to a bias towards certain tokens or sequences, even if they are unlikely or irrelevant in the given context. For example, consider a language model trained on a corpus of news articles. If the model encounters a rare or unseen word during inference, it may still assign a non-zero probability to that word due to the normalization constraint, even though the word is highly unlikely in the context of news articles.</p><p>The consequences of label bias can be significant, especially in cases where the data distribution is highly skewed or contains rare or unseen events. It can lead to the generation of nonsensical or irrelevant tokens, which can degrade the overall quality and coherence of the generated sequences. This bias can be particularly problematic in applications such as machine translation, text summarization, or dialogue systems, where accurate and context-appropriate generation is crucial.</p><h4><strong>Order Bias</strong></h4><p>Order bias occurs when the left-to-right generation order imposed by AR models is not the most natural or optimal order for the given task or data. In some cases, the data may prefer a different generation order or require considering the entire context simultaneously, rather than generating tokens sequentially.</p><p>For example, in text-to-speech synthesis, the natural flow of speech may not follow a strict left-to-right order. The pitch, intonation, or emphasis of a sentence can be determined by the context of the entire sentence, rather than the individual words. If the model is trained to generate the audio sequence in a left-to-right order, it may fail to capture these nuances, leading to an unnatural or strange-sounding output.</p><p>That’s it for now, in the next article I will elaborate a bit regarding non-AR models.</p><h3>Next chapter</h3><p><a href="https://medium.com/@peechapp/text-to-speech-models-part-3-non-autoregressive-models-hideout-f45e46a4f4bd">Part 3: Non-autoregressive models hideout 🕵️‍♂️</a></p><p>Author: <a href="https://www.linkedin.com/in/nick-ovchinnikov/">Nick Ovchinnikov</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*hjI4eNVs--dANvrBDxEC4w.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=636d8aa0932d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text-To-Speech Models. Part 1: Intro, Little Theory and Math]]></title>
            <link>https://medium.com/@peechapp/text-to-speech-models-part-1-intro-little-theory-and-math-0ffa5d3e0e3f?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ffa5d3e0e3f</guid>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[speech]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 08:54:33 GMT</pubDate>
            <atom:updated>2024-06-11T15:00:51.891Z</atom:updated>
            <content:encoded><![CDATA[<h3>Machine Learning Text-To-Speech: <strong>Intro, Little Theory and Math</strong></h3><h3><strong>Read the Full Series of ML TTS exploration</strong></h3><ul><li>👉Part 1: Intro, Little Theory and Math 📘</li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-2-autoregressive-models-world-636d8aa0932d">Part 2: Autoregressive models world 🌍</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-3-non-autoregressive-models-hideout-f45e46a4f4bd">Part 3: Non-autoregressive models hideout 🕵️‍♂️</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-4-autoregressive-and-hybrid-models-review-abeb2b5ec3ec">Part 4: Autoregressive and hybrid models review 📊</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-5-non-autoregressive-models-review-7ce8de9c9e9d">Part 5: Non-autoregressive models review📝</a></li><li><a href="https://medium.com/@peechapp/text-to-speech-models-part-6-delightfultts-implementation-and-training-ceb4c94fd5fb">Part 6: DelightfulTTS implementation and training 🛠️</a></li></ul><h3><strong>Training Text-to-Speech Models: A Journey from Zero to…</strong></h3><p>Developing a high-quality text-to-speech (TTS) system is a complex task that requires extensive training of machine learning models. While successful TTS models can revolutionize how we interact with technology, enabling natural-sounding speech synthesis for various applications, the path to achieving such models is often paved with challenges and setbacks.</p><p>Training a TTS model from scratch is a challenging process that involves numerous steps, from data preparation and preprocessing to model architecture selection, hyperparameter tuning, and iterative refinement. Even with state-of-the-art techniques and powerful computational resources, it’s not uncommon for initial training attempts to fall short of expectations, yielding suboptimal results or encountering convergence issues.</p><p><strong>Audio is a very complicated data structure, just take a look for a 1 sec waveform…</strong></p><figure><img alt="Audio is a very complicated data structure, just take a look for a 1 sec waveform…" src="https://cdn-images-1.medium.com/max/570/1*SNzcttMK3Q52aiIiyJ3EOA.gif" /><figcaption><em>Audio exhibits patterns at multiple time scales. Source: Google DeepMind </em><a href="https://deepmind.com/blog/wavenet-generative-model-raw-audio/">https://deepmind.com/blog/wavenet-generative-model-raw-audio/</a></figcaption></figure><p>By carefully analyzing the shortcomings of unsuccessful models and identifying the root causes of their underperformance, researchers and practitioners can gain invaluable insights into the intricacies of TTS model training. These lessons can then inform and refine the training process, leading to more robust and high-performing models.</p><p>In this article, we embark on a comprehensive journey, exploring the intricate world of TTS models.</p><h3><strong>How it all started?</strong></h3><p>At <a href="https://www.getpeech.com/">Peech</a>, we’re dedicated to making Text to Speech accessible to everyone — individuals and publishers alike. Our innovative technology converts web articles, e-books, and any written content into engaging audiobooks. This is beneficial for anyone who prefers listening to reading and, especially, individuals with dyslexia, ADHD, or vision impairments.</p><p>The heart of our app lies in its text-to-speech technology. I’ve recently joined the impressive startup’s team with the ambition of advancing our capabilities in AI and Machine Learning. To achieve this, we’ve embarked on a research journey focused on training models in this domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*An751O32I5o0CHZvofsTCg.jpeg" /></figure><p>I’m excited to share with you the results I’ve achieved in our research and model training. However, let’s go through some theory at first.</p><h3><strong>One-to-Many Mapping Problem in TTS</strong></h3><p>In TTS, the goal is to generate a speech waveform <strong>y</strong> from a given input text <strong>x</strong>. This can be represented as a mapping function <strong>f</strong> such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*FMqzyWCeULnSOG7A-AWYPw.png" /></figure><p>However, the mapping from text to speech is not unique, as there can be multiple valid speech outputs <strong>y</strong> for the same input text <strong>x</strong>. This is because speech is a complex signal that encodes various types of information beyond just the phonetic content, such as pitch, duration, speaker characteristics, prosody, emotion, and more.</p><p>Let’s denote these additional factors as a set of variation parameters <strong>v</strong>. Then, the mapping function <strong>f</strong> can be rewritten as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/1*d-3kks29TbDrnKK777nbFQ.png" /></figure><p>This means that for a given input text <strong>x</strong>, the speech output <strong>y</strong> can vary depending on the values of the variation parameters <strong>v</strong>.</p><p>For example, consider the input text <strong>x = “Hello, how are you?”</strong>. Depending on the variation parameters <strong>v</strong>, we can have different speech outputs:</p><p>- If <strong>v</strong> represents pitch contour, we can have a speech output with a rising pitch at the end (questioning tone) or a falling pitch (statement tone).</p><p>- If <strong>v</strong> represents speaker identity, we can have speech outputs from different speakers, each with their unique voice characteristics.</p><p>- If <strong>v</strong> represents emotion, we can have speech outputs conveying different emotions, such as happiness, sadness, or anger.</p><p>Mathematically, we can represent the variation parameters <strong>v</strong> as a vector of different factors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*Crk_LJNTaVwkM75E5OVqZQ.png" /></figure><p>The challenge in TTS is to model the mapping function <strong>f</strong> in such a way that it can generate appropriate speech outputs <strong>y</strong> for a given input text <strong>x</strong> and variation parameters <strong>v</strong>. This is known as the one-to-many mapping problem, as there can be multiple valid speech outputs for the same input text, depending on the variation factors.</p><p>There are two main approaches to modeling the specific aspects of the variation parameters <strong>v</strong>:</p><ol><li><strong>Modeling Individual Factors:</strong> In this approach, each variation factor is modeled independently, without considering the interdependencies and interactions between different factors. For example, a model might focus solely on predicting pitch contours or duration values, treating them as separate components. While this approach can capture specific aspects of the variation parameters, it fails to model the variation information in a comprehensive and systematic way. The interdependencies between different factors, such as the relationship between pitch and prosody or the influence of speaker characteristics on duration, are not accounted for.</li><li><strong>Unified Frameworks for Modeling Multiple Factors</strong>: Recent research has proposed unified frameworks that aim to model multiple variation factors simultaneously, capturing their complementary nature and enabling more expressive and faithful speech synthesis. In this approach, the TTS model is designed to jointly model and generate multiple variation factors, considering their interdependencies and interactions. For instance, a unified framework might incorporate modules for predicting pitch, duration, and speaker characteristics simultaneously, while also accounting for their mutual influences. Mathematically, this can be represented as a mapping function <strong>f</strong> that takes the input text <strong>x</strong> and generates the speech output <strong>y</strong> by considering the combined effect of multiple variation parameters <strong>v</strong>:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*Tx8pWICGZB6MBh3ZLBRPCA.png" /></figure><p>By modeling the variation parameters in a unified and comprehensive manner, these frameworks aim to capture the complex relationships between different factors, enabling more expressive and faithful speech synthesis that better reflects the nuances and variations present in natural speech.</p><p>The unified approach to modeling variation parameters in TTS systems has gained traction in recent years, as it addresses the limitations of modeling individual factors independently and enables the generation of more natural and expressive speech outputs.</p><h3>AR and NAR</h3><p>Globally, TTS models can be categorized as either autoregressive or non-autoregressive, and in the following chapters, we will review these two paradigms in depth, paving the way for a comprehensive understanding of the state-of-the-art in text-to-speech synthesis.</p><h3>Next chapter</h3><p><a href="https://medium.com/@peechapp/text-to-speech-models-part-2-autoregressive-models-world-636d8aa0932d">Part 2: Autoregressive models world 🌍</a></p><p>Author: <a href="https://www.linkedin.com/in/nick-ovchinnikov/">Nick Ovchinnikov</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*a-LEXpTCmYXuZbzaiQfMgA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ffa5d3e0e3f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Peech?]]></title>
            <link>https://medium.com/@peechapp/what-is-peech-a968628da209?source=rss-a422e61a20f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a968628da209</guid>
            <category><![CDATA[text-to-voice]]></category>
            <category><![CDATA[ai-startups]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[tts]]></category>
            <dc:creator><![CDATA[Peech]]></dc:creator>
            <pubDate>Wed, 27 Sep 2023 14:47:37 GMT</pubDate>
            <atom:updated>2023-09-27T14:52:53.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hTBRlP-ruZxGnEGnrc_9QQ.jpeg" /></figure><p>Thank you for selecting Peech — a powerful tool that effortlessly transforms any text into incredibly realistic AI-generated audio. Peech supports over 50 languages, including English, French, German, Italian, Spanish, and more. Immerse yourself in audio versions of articles, texts, PDFs, or ebooks of diverse formats.</p><h3>Using Peech for Work and Study</h3><p>We recognise the challenges that can be caused by paperwork during your academic journey or at work. This is where the Scanning feature might help. Simply tap green plus button, then select “<strong>Scan</strong>”, and capture an image of your document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zsw9vLV6M4AGmnQa6-Izwg.png" /></figure><p>You also have the option to select images from your library, enabling you to add as many as required. Rotate, adjust, and name your scanned documents as needed. Peech automatically converts them into audio format.</p><p>If you have documents saved in formats such as <strong>epub, docx, pdf or rtf </strong>on your device, simply tap the + button, select “<strong>Files</strong>,” and pick the desired document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A-GNKJiKw0SBGpLc5YtZcA.png" /></figure><p>In addition to the features above, there’s the option to import plain text.</p><p>You can also manage your added files. If you want to rename a document, go to the Library, tap on the three dots next to the file, and then select “Rename.” If you want to delete a file, simply tap “Remove” right here. “Feel free to try this out and then remove this article after reading.</p><p>To enhance document management use a <strong>tagging feature</strong>. Label your documents and articles for effective organization, treating tags as personalized playlists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AwMkUi4LcTF1ayBadUMHMA.png" /></figure><p>Recognizing that text can be consumed at varying speeds, explore the <strong>Speed Switch</strong> feature and select from 0.75x to 2.0x voiceover speeds. By increasing the pace, you can consume text up to <strong>5 times faster</strong>, significantly boosting productivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*39R1ewZj70vPcS-bxqqiHw.png" /></figure><h3>Listening to Books and Articles with Peech</h3><p>At Peech, we adore reading fascinating books. But more than reading, we love to listen to them. Import any eBook by tapping the + button and choosing a book from your device. Peech automatically divides it into equal sections, aiding progress tracking. Initiate playback by hitting the Play button and relish the experience.</p><p>Adding articles is equally simple, offering two options:</p><ol><li>Copy an article to the browser clipboard. Then, open the Peech app, and you’ll be prompted to save the article to your list.</li><li>Employ the mobile browser extension: select “<strong>Share</strong>” and then “<strong>Save to Peech</strong>”. Done! It’s in the app.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iIoycL8iHtwzAwK4KvHFwg.png" /></figure><p>Peech is more than just a tool — it’s an inspiration hub. The “<strong>Selection</strong>” section showcases an array of articles from known sources such as Medium, The Guardian, Bloomberg, BBC, The Washington Post, and HBR, among others. With over 30 distinct categories including Parenthood, Venture Capital, Marketing, Healthy Nutrition, Financial Markets, and Travel, Peech employs AI to tailor your experience. To fine-tune your feed, visit settings and opt for categories aligning with your interests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kx_My5boxX_HP6udWKv77g.png" /></figure><p>Ready to turn words into sound? Start using Peech now and amplify your textual universe with the magic of audio.<strong> </strong>Enhance your productivity, enrich your leisure time, and elevate your learning on this delightful auditory journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a968628da209" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>