<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by mllopartbsc on Medium]]></title>
        <description><![CDATA[Stories by mllopartbsc on Medium]]></description>
        <link>https://medium.com/@mllopart.bsc?source=rss-324e1ddc0da7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*vAtfXtJ3fCVqOBYz</url>
            <title>Stories by mllopartbsc on Medium</title>
            <link>https://medium.com/@mllopart.bsc?source=rss-324e1ddc0da7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:38:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mllopart.bsc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optimizing a multi-speaker TTS model for faster CPU inference — Part 3]]></title>
            <link>https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-3-8af6a5dcafe3?source=rss-324e1ddc0da7------2</link>
            <guid isPermaLink="false">https://medium.com/p/8af6a5dcafe3</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[deepsparse]]></category>
            <category><![CDATA[onnx]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[simplification]]></category>
            <dc:creator><![CDATA[mllopartbsc]]></dc:creator>
            <pubDate>Fri, 02 Aug 2024 14:43:04 GMT</pubDate>
            <atom:updated>2024-08-02T14:46:50.556Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimizing a multi-speaker TTS model for faster CPU inference — Part 3</h3><p>In this final part of our blog, I’ll explain in a bit more detail some other optimization techniques that we tried, even if unsuccessfully.</p><h3>Trying Deepsparse:</h3><p>Originally, the idea was to use Deepsparse to optimize the models for inference. However, after some successful attempts with static inputs, where the engine was complied and ran with inputs of the same size, I realized that the Engine doesn’t accept dynamic inputs. It’s a pitty because it was running as good as some of the best parallel executions with ONNX. In spite of that, Deepsparse was discarded as an option for optimization, as it would’ve been slower than other options. Nonetheless, it can be a viable option in the future, since Neural Magic has stated that they plan to implement dynamic inputs in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gwRQvOeEZAgfsVNr0JjNzA.png" /><figcaption>From the official Deepsparse site (<a href="https://docs.neuralmagic.com/details/faqs/">https://docs.neuralmagic.com/details/faqs/</a>)</figcaption></figure><h3>Does the simplifier work?</h3><p>After the success of the previous experiment, I tried using the <a href="https://github.com/daquexian/onnx-simplifier">onnx-simplifier</a> library to see if reducing the model size and redundancies I could achieve better results for inference. Here’s the command I used to simplify the model for dynamic inputs:</p><pre>onnxsim matxa_vocos_merged_HF.onnx matxa_vocos_merged_HF_simplified_dynamic.onnx --overwrite-input-shape model1_x:1,-1 model1_x_lengths:1 model1_scales:2 model1_spks:1</pre><p>And here’s the results of the simplification:</p><p><a href="https://gist.github.com/mllopartbsc/b03788c1c9c4365455607dcd15796ee1">https://gist.github.com/mllopartbsc/b03788c1c9c4365455607dcd15796ee1</a></p><p>However, inference times increased:</p><p>Total time (10 executions) (s): 220.0996 seconds</p><p>Average execution time per inference (s): 22.0100 seconds</p><p>Standard deviation of execution times (s): 3.3705 seconds</p><p>This might be due to a number of reasons:</p><ul><li><strong>Model Structure Changes</strong>: The simplification process might alter the structure of the model in ways that are not optimal for the inference engine. Simplified models can sometimes include operations or sequences of operations that are less efficient for certain hardware or software environments.</li><li><strong>Graph Optimization Limitations</strong>: The simplifier aims to reduce redundancy and simplify operations, but these changes might interfere with optimizations that the inference engine could perform. For example, the engine might have specific optimizations for certain complex operations that are lost when those operations are simplified.</li><li><strong>Memory Access Patterns</strong>: Simplification can alter memory access patterns. If the simplification changes how data is accessed and stored, it might result in less efficient use of caches and memory bandwidth.</li><li><strong>Operation Fusion and Parallelism</strong>: Some inference engines perform operation fusion (combining multiple operations into a single, more efficient operation) and parallel execution. Simplifying the model might break these fused operations or disrupt parallel execution, leading to longer inference times.</li></ul><p>However, if the limitation of the model comes from the memory usage, the simplifier library can be a good option to reduce model size.</p><h3>Does pruning work?</h3><p>As my last attempt, i tried with model pruning, which resulted in another unsuccessful optimization. Here’s the script I used for pruning:</p><p><a href="https://gist.github.com/mllopartbsc/424216addc20c9495a7c4c310fc06540">https://gist.github.com/mllopartbsc/424216addc20c9495a7c4c310fc06540</a></p><p>The reduction in model size was of about 5 Kb. However, these are the inference results:</p><p>Total time (10 executions) (s): 179.1162 seconds</p><p>Average execution time per inference (s): 17.9116 seconds</p><p>Standard deviation of execution times (s): 2.3069 seconds</p><p>They are slightly worse than the non-pruned version. Some of the reasons why these results are very similar to those from the simplifier library:</p><ul><li><strong>Irregular Sparsity</strong>: Pruning often results in sparse matrices. If the sparsity pattern is irregular, many inference engines may not be optimized to handle such sparsity efficiently, leading to increased inference times.</li><li><strong>Lack of Hardware Support</strong>: Many current hardware accelerators and inference engines are optimized for dense operations. Sparse operations, especially those with irregular sparsity, might not benefit from these optimizations and can even be slower due to the overhead of handling sparse data structures.</li><li><strong>Cache and Memory Access Patterns</strong>: In the same way as the simplified model, pruned models can have different memory access patterns compared to their dense counterparts. These new patterns can lead to suboptimal cache usage and increased memory access latency.</li><li><strong>Operation Overhead</strong>: Handling sparsity often involves additional overhead to skip zero values, which can outweigh the computational savings, particularly if the percentage of pruned weights is not high enough.</li><li><strong>Fragmented Computations</strong>: Pruning can fragment the computations, leading to less efficient execution. The benefits of parallel processing might be reduced because the computations are no longer as well-structured or evenly distributed.</li><li><strong>Graph Rewriting Overheads</strong>: Pruning changes the graph of the model. Some inference engines might struggle with the altered graph, leading to inefficient execution plans or suboptimal use of the available hardware.</li></ul><p>Here’s the link for part 1:</p><p><a href="https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-1-165908627829">https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-1-165908627829</a></p><p>And here’s the link for part 2:</p><p><a href="https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-2-fa9fc48d1635">https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-2-fa9fc48d1635</a></p><p>All the the resources developed within the Aina project (projecteaina.cat) are available at the Aina Kit (projecteaina.cat/ainakit), an open open-source repository where users can access and download the generated technologies.</p><p>This work/research has been promoted and financed by the Government of Catalonia through the Aina project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8af6a5dcafe3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing a multi-speaker TTS model for faster CPU inference — Part 2]]></title>
            <link>https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-2-fa9fc48d1635?source=rss-324e1ddc0da7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa9fc48d1635</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[onnx]]></category>
            <category><![CDATA[cpu]]></category>
            <dc:creator><![CDATA[mllopartbsc]]></dc:creator>
            <pubDate>Fri, 02 Aug 2024 14:42:16 GMT</pubDate>
            <atom:updated>2024-08-02T14:47:14.989Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimizing a multi-speaker TTS model for faster CPU inference — Part 2</h3><p>In this second part of the blog, I’ll explain in detail how we chose the previously mentioned ONNX settings for intra- and inter-operator parallelism.</p><h3>Using the parallelism settings in a smart way:</h3><p>This section contains most of the meat of this work. I’ll explain how, just by changing one setting of ONNX parallelism smartly, we can drastically reduce latency. Needless to say, this experiment turned out to be the most successful one, since it lead to the lowest RTF and latency. I’ll jump straight to it, the experiment consisted in something as simple as adding this to the session options of the ONNX runtime:</p><pre>sess_options.intra_op_num_threads = args.intra_op</pre><p>Where</p><pre>args.intra_op = 20</pre><p>Like the rest of them, this experiment was ran with 20 CPUs and these are the results:</p><p>Total time (10 executions) (s): 165.2652 seconds</p><p>Average execution time per inference (s): 16.5265 seconds</p><p>Standard deviation of execution times (s): 0.6891 seconds</p><p>RTF: 0.09</p><p>Really impressive, right?</p><p>Now, I’ll explain why we decided to set the number of intra-operator threads to match the number of CPUs available for the experiment. First of all, take a look at this results from our merged ONNX model, which shows the average number of operators per layer:</p><h3>Matxa</h3><p>Total number of nodes: 1,30E+04</p><p>Total number of layers: 9,51E+03</p><p>Average number of nodes per layer: 1,37E+00</p><h3>AlvoCat</h3><p>Total number of nodes: 2,72E+02</p><p>Total number of layers: 2,21E+02</p><p>Average number of nodes per layer: 1,23E+16</p><p>You can check this yourself by running this script:</p><p><a href="https://gist.github.com/mllopartbsc/eb73a889d3360c10f5dcc7e71241400e">https://gist.github.com/mllopartbsc/eb73a889d3360c10f5dcc7e71241400e</a></p><p>Before I continue with the explanation, let me introduce the concepts of intra and inter operator threads. In the context of neural networks(NNs), these concepts belong to parallelism strategies used to optimize the execution of NNs operations on multi-core processors. The goal, obviously, is to achieve better performance by efficiently utilizing the available computational resources.</p><h3>Intra-Operator Parallelism</h3><p>Intra-operator parallelism (also known as operator-level parallelism) involves parallelizing the execution of a single operator across multiple threads. Each operator (e.g., matrix multiplication, convolution) can be broken down into smaller tasks that can be processed simultaneously. This is particularly useful for large operations where a single operator dominates the computational cost.</p><h3>Inter-Operator Parallelism</h3><p>Inter-operator parallelism (also known as graph-level parallelism) involves running different operators concurrently on different threads. This approach takes advantage of the computational graph structure of NNs, where some operations are independent and can be executed in parallel.</p><p>To sum it up:</p><ul><li><strong>Intra-operator threads</strong>: run tasks inside an operator in parallel.</li><li><strong>Inter-operator threads</strong>: allows parallel operators to run concurrently.</li></ul><p>Now, with this information, take a look at Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*FyM67e1pLbQeUR7C" /><figcaption>Figure 1 — Examples of Synchronous Scheduling (graph left), Asynchronous Scheduling (graph middle), and changing from one(table up) to four thread pools(table down) using the same quantity of hardware resources (CPUs). In this example, there are 32 CPUs.</figcaption></figure><p>On the left side and middle of Figure 1, you can see two examples of the same neural network, presented as a graph. In the leftmost graph, the network is being ran synchronously, meaning that the computations of an operator only start when the computations of the previous operator have finished. The graph in the middle presents a different situation, where the network is ran asynchronously, meaning that operators independent to each other are executed in parallel. Then, on the right side of Figure 1 you can see a pool of 32 CPUs. The grid at the bottom right of Figure 1 is a visualization of what happens when you set the configuration for inter operator threads to 4: with a pool of 32 CPUs, now each inter-operator thread can use 8 CPUs. <br>The goal is to be able to parallelise the execution of this network as much as possible, so that each CPU doesn’t waste any time in synchronization and spends most of the time executing workload. With this goal in mind, we need to strike a balance between the number of intra and inter operator threads assigned to run the network. As you can see, the example network has an average number of operators per layer of 1.6 (8 operators / 5 layers). Given that, I’ll list some possible scenarios and wheter or not they are best suited for the task:</p><p><strong>1- To assign the number of intra-operator threads to be 32 and the number of inter-operator threads to be 1: <br></strong>This option is wrong because the average number of operators per layer is 1.6. This means that, in most layers, the number of operators is closer to 2 than to 1. Hence, If we only create intra-operator threads, the network will run synchronously and we won’t be able to parallelise the execution of operators within the same layer. In short, there would be 32 CPUs executing a single operator when you could’ve had 16 CPUs with one and 16 CPUs with the other one. Because of that, this is the wrong option. ❌</p><p><strong>2- To assign the number of intra-operator threads to be 1 and the number of inter-operator threads to be 32:</strong></p><p>This option is wrong because you’d have just one cpu running one operator and another single cpu running the remaining operator from the same layer, and 30 CPUs waiting. ❌</p><p><strong>3- To assign the number of intra-operator threads to be 32 and the number of inter-operator threads to be 32:</strong></p><p>In this case, even though you’d have the best configuration for every possible scenario, the fact that you are creating more threads than cores available will create a huge synchronization overtime. This effect is called oversubscription, and it unfortunately leads to slower processing due to the calculations needed to change threads for each scenario. ❌</p><p><strong>4- To assign the number of intra-operator threads to be 16 and the number of inter-operator threads to be 2:</strong></p><p>This is the correct answer. Since most of the time you’ll be computing a layer that has close to 2 operators, it makes sense that you use two inter-operator threads, with 16 intra-operator threads each, for a total of 32 threads. That way, you avoid the effects of oversubscription while also offering the best configuration for the most common scenario. ✅</p><p>Now, let’s go back to our own case, which would look like figure two:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*CJF4ITSiwN0s1EthNa3BWA.png" /><figcaption>Figure 2 — Simplistic Graph visualization of the Matxa + Vocos ONNX model</figcaption></figure><p>Since the average number of operators per layer for both models is &lt;1.5 , it only makes sense to run the network sequentially, that is, without using any inter operator threads. By doing this, all the computing resources can focus on performing internal operations within each layer’s nodes, which is precisely what intra-operator threads do. Knowing that, it only makes sense to assign all CPUs to intra-operator threads. By doing that, we can eliminate all the synchronization overtimes, an achieve good results like the ones that I’ve just shown. For more information about this subject, please check this paper:</p><p><a href="https://arxiv.org/abs/1908.04705">Exploiting Parallelism Opportunities with Deep Learning Frameworks</a></p><p>Another interesting insight we obtained from this experiment, was to see that adding more CPUs (and therefore more intra-operator threads) does not always lead to reduced latency. Here are the results with 40 CPUs and 60 CPUs, with the same amount of intra-operator threads:</p><p><strong>40 CPUs:</strong></p><p>Total time (10 executions) (s): 143.5563 seconds</p><p>Average execution time per inference (s): 14.3556 seconds</p><p>Standard deviation of execution times (s): 0.7709 seconds</p><p>RTF: 0.079</p><p><strong>60 CPUs:</strong></p><p>Total time (10 executions) (s): 166.9228 seconds</p><p>Average execution time per inference (s): 16.6923 seconds</p><p>Standard deviation of execution times (s): 0.2512 seconds</p><p>RTF: 0.092</p><p>As you can see, 40 CPUs does actually lead to an improvement in the performance, which makes sense because you have more compute available for internal operations within nodes. However, at 60CPUs the model performs even worse than with 20CPUs. Most likely, this is happens due to the effects of hyperthreading:</p><p>Hyperthreading can cause overhead in inference because it introduces context-switching between threads, which consumes CPU cycles and can lead to inefficiencies. Previously, we called that synchronization. Additionally, resource contention among threads can result in increased latency and suboptimal performance for inference tasks</p><p>Additionally, I will present the experiment that lead me to confirm all of the previously mentioned conclusions. For efficiency purposes, I did this experiment with a shorter sentence, which reduced the inference times. The results are in this google sheets table, which you can access with this link:</p><p><a href="https://docs.google.com/spreadsheets/d/1rx1J06yoDgDbZHFpfqTQ1FWbnstEEwLFzBBUfdp1XcU/edit?usp=sharing">Intra/Inter results</a></p><p>In the table , you can see the inference times in seconds for different configurations of the ONNX session. In the first column are the inter-operator threads, whereas in the first row are the intra-operator threads, both numbered from 1 to 20. Taking into account a standard deviation of about 0,3 seconds, you can see that the best results are with 20 intra-operator threads. Then, adding inter-operator threads leads to no effect. With these results, I formulated the reasoning presented above.</p><p>When the ONNX runtime is trying to execute the network with the default configuration, it loses latency in the process of changing thread settings. This is also called the synchronization of the threads or synchronization overtime. Because of that, it works worse than with the 20 intra-operator threads and 1 inter-operator thread. Essentially, it would be something similar to having the settings be 32 intra-operator and 32 inter-operator threads.</p><p>Hopefully, this explanation was clear enough so that you could understand why we decided to choose these settings. Again, thank you for your attention! 😊</p><p>Here’s the link for part 3:</p><p><a href="https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-3-8af6a5dcafe3">Optimizing a multi-speaker TTS model for faster CPU inference — Part 3</a></p><p>And here’s the link for part 1:</p><p><a href="https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-1-165908627829">Optimizing a multi-speaker TTS model for faster CPU inference — Part 1</a></p><p>All the the resources developed within the Aina project (projecteaina.cat) are available at the Aina Kit (projecteaina.cat/ainakit), an open open-source repository where users can access and download the generated technologies.</p><p>This work/research has been promoted and financed by the Government of Catalonia through the Aina project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa9fc48d1635" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing a multi-speaker TTS model for faster CPU inference — Part 1]]></title>
            <link>https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-1-165908627829?source=rss-324e1ddc0da7------2</link>
            <guid isPermaLink="false">https://medium.com/p/165908627829</guid>
            <category><![CDATA[onnx]]></category>
            <category><![CDATA[cpu-utilization]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <dc:creator><![CDATA[mllopartbsc]]></dc:creator>
            <pubDate>Fri, 02 Aug 2024 14:41:24 GMT</pubDate>
            <atom:updated>2024-08-02T14:48:29.046Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimizing a multi-speaker TTS model for faster CPU inference — Part 1</h3><p>Our goal was to optimize🍵 Matxa, a Catalan multispeaker and multidialectal text-to-speech (TTS) model which uses 🥑 alVoCat as a vocoder. In the end, we obtained a 4.8x speedup for our model, here’s how we did it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZmN_vamTGvUFczzs" /><figcaption>The official logo for the Matxa model</figcaption></figure><p>You can try out the combination of the models in our Huggingface demo:</p><p><a href="https://huggingface.co/spaces/projecte-aina/matxa-alvocat-tts-ca">Catalan Text-to-Speech - a Hugging Face Space by projecte-aina</a></p><p>The objective of this blog is to show how we achieved the outstanding result of a 4.8x speedup. With that objective in mind, I’ll show both our victories and our defeats. The summary of our optimization process is the following:</p><p>We initially established a baseline performance evaluation with the original checkpoints. Then, we converted the models to ONNX, which led to a 10% improvement in latency. After that, the most significant performance speedup came from optimizing the ONNX parallelism settings by aligning the number of intra-operator threads with the number of available CPUs. By doing this, we substantially reduced the synchronization overhead. We decided to take this approach because the graph of the neural network is almost non-branched, a perfect scenario for using only intra-operator threads. Similar Attempts with DeepSparse, model simplification, and pruning were counterproductive. Overall, on a <a href="https://gist.github.com/mllopartbsc/cd7215981ab8ec2caa27caa434d3991b">sample text</a> we reduced the average inference time from 79.8 to 16.5 seconds, enhancing real-time performance by 4.8x. In the table below, you can see the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GrmTupb6evf4Jgm8M11kWQ.png" /><figcaption>Table 1 — The main results of our inference experiments.</figcaption></figure><p>The contents of this study are divided in three parts:</p><ul><li>Part 1 is a summary of our work.</li><li><a href="https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-2-fa9fc48d1635">Part 2</a> delves into the specifics of ONNX intra- and inter-operator parallelism.</li><li><a href="https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-3-8af6a5dcafe3">Part 3</a> narrates our compression and simplification efforts.</li></ul><p>Each experiment ran the <a href="https://gist.github.com/mllopartbsc/cd7215981ab8ec2caa27caa434d3991b">sentence’s</a> inference 10 times. Running an inference experiment on CPUs multiple times is essential to ensure reliable and accurate performance measurements, as it helps mitigate the effects of measurement noise, CPU performance variability, system state, resource contention, caching effects, and outliers. As you will see, I also present measurements for standard deviation together with each result, so that variability can also be taken into account.</p><p>I will be using Github Gists to share some of the code that I used for this project. In regards to the system specifications, all experiments have been carried out with 20 Intel Xeon Platinum 8480+ 56C 2GHz CPUs.</p><p>Here’s the index for this blog:</p><p>0- Introduction to the architectures</p><p>1- The raw models</p><p>2- The conversion to ONNX</p><p>3- Changing ONNX’s parallelism settings</p><p>4- Trying other optimization techinques</p><p>5- Future Work</p><p>6- Conclusion</p><h3>0- Introduction to the architectures:</h3><p>🍵Matxa and 🥑alVoCat are based on Matcha-TTS and Vocos architectures respectively:</p><p><strong>· Matcha-TTS</strong> is a non-autorregressive encoder-decoder model designed for fast acoustic modelling in TTS. The encoder part processes input sequences of phonemes and, together with a phoneme duration predictor, outputs averaged acoustic features. And the decoder, which is essentially a U-Net backbone based on the Transfomer architecture, predicts the refined spectrogram. The model is trained with optimal-transport conditional flow matching. This yields an ODE-based decoder capable of generating high output quality in fewer synthesis steps.</p><p><strong>· Vocos</strong> is a fast neural vocoder designed to synthesize audio waveforms from acoustic features. Unlike other typical GAN-based vocoders, Vocos does not model audio samples in the time domain. Instead, it generates spectral coefficients, facilitating rapid audio reconstruction through inverse Fourier transform. The goal of this model is to provide an alternative to hifi-gan that is faster and compatible with the acoustic output of several TTS models.</p><h3>1- The raw models:</h3><p>This test serves as a baseline to check the performance of the models without any optimizations applied to them. It uses the checkpoints for Matxa and alVoCat which you can find <a href="https://huggingface.co/projecte-aina/matxa-tts-cat-multiaccent/blob/main/checkpoint_epoch%3D2399.ckpt">here</a>. The models were exported with a timestep of 10 to reduce latency.</p><p>This is the script that I used to run the model for inference 10 times and obtain latency results:</p><p><a href="https://gist.github.com/mllopartbsc/a179c2ac7a71c2ae056bcd5025415eac">https://gist.github.com/mllopartbsc/a179c2ac7a71c2ae056bcd5025415eac</a></p><p>And these are the results:</p><p>Total time (10 executions)(s): 798.1826 seconds</p><p>Average execution time per inference(s): 79.8183 seconds</p><p>Standard deviation of execution times(s): 0.5585 seconds</p><p>Real-Time Factor(RTF): 0.44</p><p>These are some promising results to start with, but we want to be able to lower latency as much as possible, so this is what we tried:</p><h3>2- The conversion to ONNX:</h3><p>The second experiment that we tried was to convert the models to ONNX. What we did was to first convert the Matxa and AlvoCat models to ONNX separately and then merge them. Here’s how I did it:</p><p>First of all, you should download the Matxa and AlvoCat models from this Huggingface page:</p><p><a href="https://huggingface.co/collections/projecte-aina/speech-models-656ef7581f7c43caa392c86a">SPEECH_Models - a projecte-aina Collection</a></p><p>Then, you can use this script to merge the two models together:</p><p><a href="https://gist.github.com/mllopartbsc/8032fce890fc1fe495188f952e5dfcfe">https://gist.github.com/mllopartbsc/8032fce890fc1fe495188f952e5dfcfe</a></p><p>After that, I did the inference of the models from the branch <a href="https://github.com/langtech-bsc/Matcha-TTS/tree/dev-cat">https://github.com/langtech-bsc/Matcha-TTS/tree/dev-cat</a> by creating this Python script on the matcha/onnx folder:</p><p><a href="https://gist.github.com/mllopartbsc/fa4db847fe7b0cbc1c7fbb92d5e69e88">https://gist.github.com/mllopartbsc/fa4db847fe7b0cbc1c7fbb92d5e69e88</a></p><p>Which can be executed with this command:</p><pre>python3 -m matcha.onnx.modified_infer_deepsparse_merged matxa_vocos_merged_HF_simplified.onnx --text &quot;Molt bé, gràcies Martí. En què et puc ajudar?&quot; --vocoder_path vocos_mel_spec_22khz_cat.onnx --vocoder_name &#39;vocos_mel_spec_22khz_cat.onnx&#39; --output-dir &quot;/home/mllopart/PycharmProjects/matcha_hifigan/matcha_hifigan/Matcha-TTS/outputs”</pre><p>For this test, the session options were the default ones, without any configurations for intra- or inter-operator parallelism. Here are the results:</p><p>Total time (10 executions) (s): 712.4786 seconds</p><p>Average execution time per inference (s): 71.2479 seconds</p><p>Standard deviation of execution times (s): 0.8797 seconds</p><p>RTF: 0.39</p><p>As you can see, the ONNX runtime offers a significative improvement w.r.t the baseline Pytorch runtime. This time, the inference for both ONNX models merged leads to an RTF of 0.39 in comparison to the 0.44 we got with the baseline results. It’s really impressive that just by converting the models to ONNX it leads to a 10% improvement in latency.</p><h3>3- Changing ONNX’s parallelism settings</h3><p>The most impressive aspect from this work is that, just by smartly changing one setting of ONNX parallelism, we can drastically reduce latency. Needless to say, this experiment turned out to be the most successful one, since it lead to the lowest RTF and latency. Jumping straight to it, the experiment consisted in something as simple as adding this to the session options of the ONNX runtime:</p><pre>sess_options.intra_op_num_threads = args.intra_op</pre><p>Where</p><pre>args.intra_op = 20</pre><p>Like the rest of them, this experiment was ran with 20 CPUs and these are the results:</p><p>Total time (10 executions) (s): 165.2652 seconds</p><p>Average execution time per inference (s): 16.5265 seconds</p><p>Standard deviation of execution times (s): 0.6891 seconds</p><p>RTF: 0.09</p><p>Really impressive, right?</p><p>In Part 2 of this work I’ll explain why we decided to use these settings.</p><h3>4 - Trying other optimization techniques</h3><p>Attempts to optimize models for inference using DeepSparse, ONNX simplification, and pruning yielded mixed results. Initially, DeepSparse showed promise with static inputs, but its inability to handle dynamic inputs led to its dismissal despite future plans for support. Simplifying the ONNX model with the onnx-simplifier library increased inference times, likely due to changes in model structure, optimization limitations, and memory access patterns. Similarly, pruning the model resulted in only minor size reductions and slightly worse performance, possibly due to irregular sparsity, lack of hardware support for sparse operations, and inefficiencies in memory access and computation. While these techniques did not improve performance, they highlighted areas for potential future enhancements.</p><h3>5- Future work</h3><p>On the one hand, given that Deepsparse is optimized for sparse operations, it might be a good idea to use it for inference with pruned models, when they implement dynamic inputs.</p><p>On the other hand, it can be a good idea to try implementing static quantization, a process where the weights and activations of a neural network are converted from high-precision (e.g., 32-bit floating point) to lower-precision (e.g., 8-bit integer) representations before inference, using calibration data to determine the appropriate scaling factors. By using calibration data, some of the overtime effects cause by the reasons mentioned in the pruning and simplifier sections, can be mitigated.</p><h3>6- Conclusion</h3><p>With the current optimizations, we can go from an average time of 79.8183 seconds to synthesize an audio that’s 181.80 seconds long, which gives us a real-time factor (RTF) of 0.44, to an average of 16.5265 seconds, which gives us an RTF of 0.09. These are excellent results that show a 4.8x improvement in the inference latency of our models. The insights of this work can be applied to many other optimization efforts with NNs.</p><p>Thank you for reading and for your attention! 😊</p><p>Here’s the link for part 2:</p><p><a href="https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-2-fa9fc48d1635">Optimizing a multi-speaker TTS model for faster CPU inference — Part 2</a></p><p>And here’s the link for part 3:</p><p><a href="https://medium.com/@mllopart.bsc/optimizing-a-multi-speaker-tts-model-for-faster-cpu-inference-part-3-8af6a5dcafe3">Optimizing a multi-speaker TTS model for faster CPU inference — Part 3</a></p><p>All the the resources developed within the Aina project (projecteaina.cat) are available at the Aina Kit (projecteaina.cat/ainakit), an open open-source repository where users can access and download the generated technologies.</p><p>This work/research has been promoted and financed by the Government of Catalonia through the Aina project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=165908627829" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>