<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dr. Rashmi Kethireddy on Medium]]></title>
        <description><![CDATA[Stories by Dr. Rashmi Kethireddy on Medium]]></description>
        <link>https://medium.com/@rashmi.kethireddy.research?source=rss-81b0abe7d3e2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CFb5Fj9bSeRxLOHlZdfxiA.jpeg</url>
            <title>Stories by Dr. Rashmi Kethireddy on Medium</title>
            <link>https://medium.com/@rashmi.kethireddy.research?source=rss-81b0abe7d3e2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:27:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rashmi.kethireddy.research/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Retentive Network: A Successor to Transformer for LLMs]]></title>
            <link>https://medium.com/@rashmi.kethireddy.research/retentive-network-a-successor-to-transformer-for-llms-fc2214d9686f?source=rss-81b0abe7d3e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc2214d9686f</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[retnet]]></category>
            <dc:creator><![CDATA[Dr. Rashmi Kethireddy]]></dc:creator>
            <pubDate>Wed, 09 Aug 2023 10:38:54 GMT</pubDate>
            <atom:updated>2023-08-09T10:38:54.813Z</atom:updated>
            <content:encoded><![CDATA[<p>A successor to transformers!!! Let’s understand what makes it successor to transformers.</p><p>We all know that there are several advantages of Transformers over Recurrent Neural Network (RNNs). They are:</p><ol><li>Transformers can process all elements of the sequence in parallel, thus making training faster. In contrast, RNNs must process each token sequentially.</li><li>They can operate over longer distances, while the forget gate of LSTMs (a variant of RNNs) discards information as it moves forward, causing vanishing gradients during backpropagation.</li><li>The information about previous tokens is summarized and compressed into a single state vector, which is then passed along the sequence in RNNs. The single state vector represents the information about the entire sequence in RNNs, while transformers directly attend to past tokens.</li></ol><p>However, there are disadvantages of transformers, the computation complexity of multi-head self-attention is quadratic with respect to sequence length, limiting the capability of processing long documents.</p><h3>Bottleneck operation during inference in transformers:</h3><p>The autoregressive inference of the transformer’s decoder would need to consider all n tokens. The inference for each time step depends on the multi-head attention, which has a computational complexity of O(n²). The bottleneck of the multi-head attention operation in the base transformer is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*MfCISPQNWtnlWqc-hjP0pw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/1*Es_VkH7iu8XDcKeyxcT7Gw.png" /></figure><p>Converting the parallelization of transformers to a recurrent approach would result in higher complexity during training, whereas the current parallelization of transformers leads to high computational complexity during inference. As a solution, the multi-scale retention (MSR) module of Retentive Networks takes on a dual form — both parallel and recurrent. This dual nature enables the model to perform actions in parallel during training and switch to a recurrent mode for performing actions during inference.</p><p>The dual representations of Retentive Networks are depicted in the following block diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*AtMNF95plzd2zFZFn-8Svw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*-GZlbT-sWs27jsJhHmJEtQ.png" /><figcaption><strong>Parallel operation</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*V1nz_D9qiHe5LQoAH_gnLg.png" /><figcaption><strong>Recurrent operation</strong></figcaption></figure><h3>RetNet vs Transformer:</h3><p>The inference cost of RetNet is length-invariant. For a 7B model and an 8k sequence length, RetNet decodes 8.4× faster and saves 70% of memory compared to Transformers with key-value caches. During training, RetNet also achieves 25–50% memory savings and 7× acceleration compared to the standard Transformer, with an advantage towards the highly-optimized FlashAttention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*5UR9xUSRJ936EIEZRdjfPg.png" /></figure><p>From the table, it can also be observed that RetNet perform equally well when compared to Transformers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*HzQqAxNC5sOHo0IfOWsNKw.png" /></figure><p>The intriguing properties of RetNet could make it a strong successor to Transformer for large language models.</p><p><strong>Advantages:</strong></p><ol><li>The dual principle of RetNet helps to reduce the computation time during inference while not compromising during training.</li><li>Inference cost is length invariant. This helps to easily process longer sequences.</li></ol><h3>References:</h3><ol><li><a href="https://arxiv.org/abs/2203.07852">https://arxiv.org/abs/2203.07852</a></li><li><a href="https://arxiv.org/pdf/2307.08621.pdf">https://arxiv.org/pdf/2307.08621.pdf</a></li></ol><p><strong>Stay tuned!! Proof for dual form will be updated soon.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc2214d9686f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wav2Accent: Automatic accent classification system]]></title>
            <link>https://medium.com/@rashmi.kethireddy.research/wav2accent-automatic-accent-classification-system-30f2ec57c2b3?source=rss-81b0abe7d3e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/30f2ec57c2b3</guid>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[accents]]></category>
            <category><![CDATA[deep-neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[speech]]></category>
            <dc:creator><![CDATA[Dr. Rashmi Kethireddy]]></dc:creator>
            <pubDate>Mon, 19 Sep 2022 09:51:25 GMT</pubDate>
            <atom:updated>2022-09-19T10:24:10.281Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This article is based on <strong>Learning filterbanks from raw waveform for accent classification</strong> published in <strong>International Joint Conference on Neural Networks (IJCNN). IEEE</strong>, 2020.</blockquote><h3>Why accent classification?</h3><p>The speech of speakers belonging to a particular region exhibit some similar patterns in pronunciations that are different from other areas. These dissimilarities in the patterns of pronunciations due to socio-regional influence of first language of the speaker are termed as accents/dialects of speech. Dialect is the super-class of accents which further includes vocabulary and grammatical variations. This article focuses on accent, i.e., pronunciation variations.</p><p>With the widespread use of speech recognition, speaker recognition or speaker verification systems in day to day life, racial/regional/cultural/linguistic disparities by the recognizer could be harder for a group of people to benefit from these systems [1]. So, the research is required in this direction to reduce the disparities by these systems. Humans with prior knowledge of accent can transcribe the speech better. So, including accent information can benefit these systems.</p><blockquote>Accent of speaker can be used to select accent specific automatic speech recognition (ASR) system. Accent information can be used to improve the performance of multi-accented ASR system. Accent information can also be used for automatic speaker profiling.</blockquote><h3>Proposed approach</h3><p>Most applications in speech follow two step process: one for extraction of features and the other for modelling. They usually use log mel filterbank energies/mel-frequency cepstral coefficients (MFCC) extracted from speech signal. In MFCCs, spectrum is multiplied to mel-scale that emphasizes vocal tract characteristics. But in accent classification, mel-scale distribution of filters may not always be the best representations, e.g., pitch accented languages where the emphasis should also be on vocal source information. Motivated by this, end-to-end classification of accents directly from waveforms which will reduce the effort of designing features specific to each corpus is proposed for accent classification. The convolution neural network (CNN) model architecture is designed in such a way that the initial layers exhibit similar operation as in MFCC by initializing the weights using time approximate of MFCC. The entire network along with initial layers is trained to learn accent classification.</p><h4><strong>Model architecture:</strong></h4><p>This section describes the complete architecture of the neural network. The left side of the network in Figure 1 represents approximation of log mel filterbank energies termed as trainable filterbank network (TFN), while the right side represents the other layers of the network for classifying the accents. Each layer is defined by aF-bK-cS, where a: number of filters, b: size of the kernel, and c: stride along the time axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*xf5LcvmClELtDwPDNIJuYw.png" /><figcaption><strong>Figure 1: Raw waveform CNN network architecture for accent classification.</strong></figcaption></figure><p><strong>MFCC approximated CNN layers:</strong><br>This section gives the overview of initial layers of CNN which acts as learnable replacement to log mel filterbank energies, referred as trainable filterbank network (TFN). It consists of three 1-dimensional CNN layers, one L2 pooling layer, one instance normalization layer and two operations (absolute and log operations). These layers are structured such that after an appropriate weight initialization, these acts as a replacement to MFCC. The structural details of each layer are given below:</p><p>1. First layer: CNN filter operation in this layer is equivalent to pre-emphasis operation in signal processing approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*js82P2aQbLy-KEGPqRJwsw.png" /></figure><p>where <strong>W</strong> is the weight row vector of convolution filter (size= 1 x 2), Xᵗ represents the input column vector [xᵗ−¹ , xᵗ] (which represents the previous and present input along the sequence), and yᵗ represents the output at time t along time sequence.</p><p>2. Second layer: This is a complex convolution layer with a filter size 200, and 80 such filters (40 representing real and other 40 representing imaginary part) which are initialized to complex Gabor wavelets approximating to MFCC.</p><p>3. L2 pooling is performed which is the approximation of modulus operation, which computes the magnitude of the output from real and imaginary filters, and reduces 80 filters to 40 filters.</p><p>4. Third layer: This CNN filter acts as the square of the Hanning window with a width of 25 ms (which is equivalent to 200 samples along the time axis for 8 kHz as sampling frequency) and a stride variant of 80. There are 40 such filters in this layer.</p><p>5. Since the weights are not constrained to have positive values, a log compression over the one added to the absolute of the output of the previous layer is computed.</p><p>6. Then finally an instance normalization is applied over the log compressed output to stabilize training, which is similar to mean-variance normalization.</p><p>Biases in all these three convolution layers are set to zero, to have a similar structure as in the first order scattering transform.</p><p><strong>Weight initialization</strong><br>The details of the weight initialization of CNN filters in three layers of TFN are given below:</p><p>1. First layer: The weights of CNN filters ‘W’ in the first layer are initialized to <em>[-0.97 ,1]</em> which makes this layer’s computation equivalent to the pre- emphasize operation of the speech signal.</p><p>2. Second layer: Weights are initialized with three approximations i.e., mel-scaled filterbank spectral coefficients, linear-scaled filterbank spectral coefficients, and random approximations.</p><p>For the first two variants, time domain approximates of filters are computed based on the first order scattering transform [3]. In the first variant, the filters are initialized to the approximate of MFCC. So, the center frequencies of triangular filters should be spaced based on mel-scale and the variance of each filter is computed from FWHM of triangular filter. Second variant of initialization is approximating time domain filters to linearly spaced filters in frequency domain. Third variant is random initialization of weights in CNN filters [3].</p><p>3. Third layer: Weight initialization of CNN filters in third layer is computed as the square of Hanning window [3].<br><br><strong>Other layers of the network:</strong><br>The second part of the network is learned to distinguish accents, termed as classification network. It is designed using two 1-dimensional CNN layers followed by three fully connected (FC) layers. Two one-dimensional CNN layers (500F-5K-1S and 3000F-1K-1S) covers a span of 5 frames with a stride of 1. Global averaging is done to get a fixed-length output of size 3000 passed through three fully connected (FC) layers with output 1500, 600, and 8. Finally, the log softmax layer gives a normalized estimated probability.</p><h3><strong>Speech corpus</strong></h3><p>Common voice corpus (version 1) is collected using crowd-sourcing from the people across the world [2]. Along with speech clips of the user, some metadata of the speaker is collected which includes accent. It is the read speech with a sampling frequency of 48 kHz. The data collected has 16 accents in English, namely: United States English (US), Australian English (AU), England English (EN), Canadian English (CA), Filipino English (FP), Hong kong English (HK), India and South Asia (IN), Irish English (IR), Malaysian English (ML), New Zealand English (NZ), Scottish English (SE), Singaporean English (SG), South Atlantic (Falkland Islands, Saint Helena) English (SA), Southern African (South Africa, Zimbabwe, Namibia) English (SAF), Welsh English (WE), and West Indies and Bermuda (Bahamas, Bermuda, Jamaica, Trinidad) English (WI). Considering only the speech utterances for which accent is provided and the utterances with no down votes, resulted in a subset of the dataset with imbalanced distribution of samples across accents. So, only the top frequently occurred 8 accents (SA, AU, CA, EN, IN, NZ, SE, and US) are considered which resulted in 57356 utterances in train set, 1200 utterances in validation (val.) set, and 1175 utterances in test set. The average length of utterance is 4.12 seconds.</p><h3>Results and Discussion</h3><p><strong>Analysis of CNN filters:</strong></p><p>Figure 2 shows the heat-map of the magnitude of frequency responses for the filters for different weight initializations. Figure 2 (a) shows the frequency responses of network with time domain filters approximated to mel-scaled filterbanks. The weights of TFN part is fixed for Figure 2(a), while weights of classification part are learnt during training, so the distribution of filters from TFN remained intact. Figure 2 (b), (c), and (d) shows the frequency responses of network with time domain filters approximated to mel-scaled filterbanks, linear-scaled filterbanks, and random initializations, respectively. For figures 2 (b), (c), and (d), along with classification part of network, TFN is also trained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qV-lQ6IEKqhVs88G5VN3AA.png" /></figure><figure><img alt="Magnitude of frequency response of 40 filters (modulus of complex filters) ordered by their peak amplitudes. (a) initialized to time approximated MFSC, (b) learned filters which were initialized to time approximated MFSC initialization, © learned filters which were initialized to time approximate of linearly places filterbanks, and (d) learned filters which were initialized randomly." src="https://cdn-images-1.medium.com/max/1024/1*rWMG1BeFKZCWxaVCp-00kQ.png" /><figcaption><strong>Figure 2: Magnitude of frequency response of 40 filters (modulus of complex filters) ordered by their peak amplitudes. (a) initialized to time approximated MFCC, (b) learned filters which were initialized to time approximated MFCC initialization, © learned filters which were initialized to time approximate of linearly places filterbanks, and (d) learned filters which were initialized randomly.</strong></figcaption></figure><p>For accent classification, the distribution of filters after learning is distorted where filters 1 to 25 are arranged below 1 kHz giving much more emphasis to the low frequency components and other filters (25 to 40) are arranged with a very steep linear scale (in Figure 2 (b)), which supported our initial hypothesis. Even though filters are analytically initialized, a moderate amount of energy is leaked into negative frequencies after learning, resulting in symmetric energies for low frequencies below 1 kHz. The heat-maps of filters in Figure 2 (c ) and (d) represents filters learnt from randomly initialized and time approximates of linearly placed filters. All the variants of initialization showed a similar distribution after learning, however training took more epochs for randomly initialized weights and time approximates of linear scaled filters</p><p><strong>Analysis of loss:</strong></p><p>It is observed that training with fixed filters (fixing the weights of TFN to pre-initialized) is better than learnt filters with random initialization. Further, learnt filters with weight initialization is better than fixed filters (fixing the weights of TFN to pre-initialized) for accent classification. These two observations showed the importance of learning filterbanks and appropriate initialization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*WYxHvMDSSCdp-cAyH-NUbA.png" /><figcaption><strong>Figure 3: Loss convergence of random initialization vs initialization to MFCC approximated filters.</strong></figcaption></figure><p>From Figure 3, it can be observed that the network with random initialized weights (green color) converged slowly when compared to MFCC approximated time domain (TD) filters (red color), which highlights the importance of initialization of CNN filters to MFCC approximated filters.</p><h3>Conclusion</h3><p>Our initial hypothesis was that hand engineering features for accent classification may need domain knowledge and may not perform up to the mark. So, this study mainly explored learning filterbanks which are initialized based on hand-designed features which are embedded as part of CNN network for accent classification. The learned filters changed in their distribution and bandwidths. It is observed that these learned filters improved the performance when compared to fixed filters and hand-engineered features which supported our hypothesis. It is also found that the efficient initialization of filterbanks will converge the network faster.</p><h3>References:</h3><p>[1] <a href="https://www.pnas.org/doi/10.1073/pnas.1915768117">https://www.pnas.org/doi/10.1073/pnas.1915768117</a></p><p>[2] Mozilla, “Project Common Voice,[online],” Available: <a href="https://voice./">https://voice.</a><br>mozilla.org/en/data, 2017</p><p>[3] R. Kethireddy, S. R. Kadiri and S. V. Gangashetty, “Learning Filterbanks from Raw Waveform for Accent Classification,” 2020 International Joint Conference on Neural Networks (IJCNN), 2020, pp. 1–6, doi: 10.1109/IJCNN48605.2020.9206778.</p><p>[4] N. Zeghidour, N. Usunier, G. Synnaeve, R. Collobert, and E. Dupoux,<br>“End-to-end speech recognition from the raw waveform,” in Proc.<br>Interspeech, 2018, pp. 781–785.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30f2ec57c2b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>