<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vigneswaran C on Medium]]></title>
        <description><![CDATA[Stories by Vigneswaran C on Medium]]></description>
        <link>https://medium.com/@120003362?source=rss-34816a7d6835------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*c4lHvgCbXj5cp_rn6AlmQw.jpeg</url>
            <title>Stories by Vigneswaran C on Medium</title>
            <link>https://medium.com/@120003362?source=rss-34816a7d6835------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:19:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@120003362/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Task-Optimized Neural Network Replicates Human
Auditory Behavior, Predicts Brain Responses, and…]]></title>
            <link>https://medium.com/read-a-paper/a-task-optimized-neural-network-replicates-human-auditory-behavior-predicts-brain-responses-and-3f105ae7f2d1?source=rss-34816a7d6835------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f105ae7f2d1</guid>
            <category><![CDATA[human-brain]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Vigneswaran C]]></dc:creator>
            <pubDate>Wed, 02 Dec 2020 04:02:15 GMT</pubDate>
            <atom:updated>2020-12-02T04:02:15.018Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>A Task-Optimized Neural Network Replicates Human<br>Auditory Behavior, Predicts Brain Responses, and<br>Reveals a Cortical Processing Hierarchy — Read a Paper</strong></h4><p><a href="https://www.cell.com/neuron/fulltext/S0896-6273(18)30250-2?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627318302502%3Fshowall%3Dtrue">A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KudIfN2hD5uiakMl" /><figcaption>Photo by <a href="https://unsplash.com/@adigold1?utm_source=medium&amp;utm_medium=referral">Adi Goldstein</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Summary — </strong>Used Hierarchical Neural Networks to mimic the auditory model of human brain and reported many interesting observations. Multitasking network for both speech and music recognition is designed, and the behavioral error trends of the network is compared with human volunteers. Task specific branching after shared layers in the network architecture supports the plausible organization of the auditory block.</p><h4>Experimental Setup</h4><p>Two auditory-based tasks were chosen for the work, 1. <em>Word recognition</em>: identify the listened word and 2) <em>Music genre prediction</em>: identify the genre of the music played. To add robustness, a reasonable amount of noise is added to the data. Now instead of having two separate task-specific models, a single branched network with initial shared layers had branched out task specific heads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*WdhPh0fgHbKJ_u2ontzLtw.png" /><figcaption>Network Architecture [<a href="https://www.cell.com/neuron/fulltext/S0896-6273(18)30250-2?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627318302502%3Fshowall%3Dtrue">Source</a>]</figcaption></figure><p>With this trained model optimized to solve both the tasks, the exciting takeaways are,</p><h4><strong>Task independent branch points</strong></h4><p>Patterns when varying the selection of layer to branch out for both tasks showed similar behavior. <em>Branch point at layer 3 was optimal.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*YN_8dABJnJRph5smzB15UA.png" /><figcaption>Performance when varying branch point [<a href="https://www.cell.com/neuron/fulltext/S0896-6273(18)30250-2?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627318302502%3Fshowall%3Dtrue">Source</a>]</figcaption></figure><h4><strong>Human vs Network</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YLHFFpIjEQvn5_hcXAfNBQ.png" /><figcaption>Proportion correct for different words in human and network [<a href="https://www.cell.com/neuron/fulltext/S0896-6273(18)30250-2?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627318302502%3Fshowall%3Dtrue">Source</a>]</figcaption></figure><p>Human volunteers and trained network were given the same tasks of recognizing different types of words and to recognize genres of musics played with varying noise levels. In case of word detection, clean signals had more accuracy and words of musical tone were easier to recognize. For both the tasks there was a clear positive correlation between humans and network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*unpQJ_YymeOMBSe3lAJpRA.png" /><figcaption>Confusion matrices of prediction of music genres in human and network [<a href="https://www.cell.com/neuron/fulltext/S0896-6273(18)30250-2?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627318302502%3Fshowall%3Dtrue">Source</a>]</figcaption></figure><h4>Voxel responses</h4><p>To confirm the behavioral errors encountered in the human subjects has matching voxel responses in the attributed parts of the brain, <a href="https://en.wikipedia.org/wiki/Functional_magnetic_resonance_imaging">fMRI</a> scan was used to record the active brain regions during the task. The observations were consistent with the hypothesis and there were clear responses in the plausibly attributed parts of the brain. Noticing that the non-primary auditory cortex had little response to outliers indicate that the apparent hierarchical structure is general enough to filter input based on salience and compute activation, but not a simple neural selectivity part for speech and music.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*CvfN62zs6c8dIBv1qvaPPA.png" /><figcaption>Different feature selection performances for different models and corresponding region of interests in human brain [<a href="https://www.cell.com/neuron/fulltext/S0896-6273(18)30250-2?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS0896627318302502%3Fshowall%3Dtrue">Source</a>]</figcaption></figure><p>Optimizing single architecture for both tasks with specialized branches and replicating the behavioral errors as humans showed that the model shall be treated as a quantitative model of human cortical organization, and also hints a completely different generalization ability of <em>multitasking</em> in deep learning models.</p><p>Hope you enjoyed reading (<em>Espero que disfrutado</em>) ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f105ae7f2d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/read-a-paper/a-task-optimized-neural-network-replicates-human-auditory-behavior-predicts-brain-responses-and-3f105ae7f2d1">A Task-Optimized Neural Network Replicates Human
Auditory Behavior, Predicts Brain Responses, and…</a> was originally published in <a href="https://medium.com/read-a-paper">Read a Paper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generalization in multitask deep neural classifiers: a statistical physics approach — Read a paper]]></title>
            <link>https://medium.com/read-a-paper/generalization-in-multitask-deep-neural-classifiers-a-statistical-physics-approach-read-a-paper-cd525b6be066?source=rss-34816a7d6835------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd525b6be066</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[multitask]]></category>
            <dc:creator><![CDATA[Vigneswaran C]]></dc:creator>
            <pubDate>Sat, 07 Nov 2020 07:37:35 GMT</pubDate>
            <atom:updated>2020-11-07T07:37:35.568Z</atom:updated>
            <content:encoded><![CDATA[<h3>Generalization in multitask deep neural classifiers: a statistical physics approach — Read a paper</h3><p><a href="https://arxiv.org/abs/1910.13593">Generalization in multitask deep neural classifiers: a statistical physics approach</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*8sodF6iNbWueraxydLPZkQ.gif" /><figcaption>Source: [<a href="https://giphy.com/gifs/Nx85vtTY70T3W">Giphy GIF</a>]</figcaption></figure><p>Training multitask deep learning models, has become an increased interest to the scientific community, as it ensures, that the deep learning models shall be extended to more generalized and intuitive systems. The most prevalent method of training multitask model is to add an auxiliary task to the primary task, such that the model should able to collect the optimal features to solve both the objectives simultaneously. However, there has been no study, clearly stating how should the main primary task should be “related” to other auxiliary tasks to gain improved multitasking benefit. In this work, the authors gives a succinct explanation on quantifying the “relatedness” with valid empirical and experimental underpinnings.</p><h4>Student-Teacher Networks</h4><p>During training, the teacher gives out noisy vector outputs (by adding noise to the network weights). Whereas training and testing in the student network is tested against the noise-free labels. In multitask setup, the student network is stacked with shared hidden layers followed by task specialized branches. The loss function is set as weighted sum of two tasks (primary and auxiliary task) <em>loss_overall = Wa * loss_a + Wb * loss_b (Wa and Wb are weights for loss_a and loss_b respectively).</em></p><h4>Multitask Dynamics</h4><p>In [Ref1], the authors proposed the task-relatedness in multitask learning as the function of angles between the singular vectors of the implicit learned mapping by the network. (Note: you can refer <a href="https://web.mit.edu/be.400/www/SVD/Singular_Value_Decomposition.htm">this link</a> to know about Singular Value Decomposition and singular vector). Picking up the main task as “A” and auxiliary task as “B”, whose main aim is to improve the performance on task “A”, the relatedness between them shall be written as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*lvudcBWAkmAnAI2d6pBPPQ.png" /><figcaption><a href="https://arxiv.org/abs/1910.13593">Source</a></figcaption></figure><p>and W_A an W_B are weight matrices of teacher network for task A and B respectively, and their SVDs are,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/1*Lr7NOrc5tLlH4lJics5h8A.png" /><figcaption><a href="https://arxiv.org/abs/1910.13593">Source</a></figcaption></figure><h4>Multitask Benefit</h4><ol><li>Increase in relatedness and Signal to Noise ratio (SNR) directly improves the multitask benefit</li><li>Similarly, unrelated and and less SNR deteriorates the learning</li><li>Before the multitask learning, a base performance with main task shall help to improve the multitask benefit</li></ol><p>SNR is increased with increase in singular values for teacher A (from 0.1 to 10), and multitask benefit is directly correlated with task relatedness and SNR for related task. Similarly, multitask benefit is also contingent upon the training points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2fHw6i1U8IInCTByDzdD5g.png" /><figcaption>[<a href="https://arxiv.org/abs/1910.13593">Source</a>]</figcaption></figure><p>With increase in amount of auxiliary task data points and relatedness along with high SNR main task, gives maximum multitask benefit. Also, increasing the non-linear student network depth also affects the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ed7R9eTjORee1VHl5QkucA.png" /><figcaption>[<a href="https://arxiv.org/abs/1910.13593">Source</a>]</figcaption></figure><p>The work proves its claims of multitask benefit with proper empirical and experimental results. Defining the concept of task relatedness, drawing them in theoretical sense using dynamics of learning and figuring out the factors that affect are the highlights of the work.</p><p>Hope you enjoyed reading :)</p><h4>Reference</h4><ol><li><em>Lampinen, Andrew K., and Surya Ganguli. “An analytic theory of generalization dynamics and transfer learning in deep linear networks.” arXiv preprint arXiv:1809.10374 (2018).</em></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd525b6be066" width="1" height="1" alt=""><hr><p><a href="https://medium.com/read-a-paper/generalization-in-multitask-deep-neural-classifiers-a-statistical-physics-approach-read-a-paper-cd525b6be066">Generalization in multitask deep neural classifiers: a statistical physics approach — Read a paper</a> was originally published in <a href="https://medium.com/read-a-paper">Read a Paper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reverse engineering RNN reveals line attractor dynamics — Read a Paper]]></title>
            <link>https://medium.com/read-a-paper/reverse-engineering-rnn-reveals-line-attractor-dynamics-read-a-paper-cae3e3f8638b?source=rss-34816a7d6835------2</link>
            <guid isPermaLink="false">https://medium.com/p/cae3e3f8638b</guid>
            <category><![CDATA[neurips-2019]]></category>
            <category><![CDATA[interpretability]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[system-dynamics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vigneswaran C]]></dc:creator>
            <pubDate>Thu, 15 Oct 2020 07:01:50 GMT</pubDate>
            <atom:updated>2020-10-15T10:59:12.918Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reverse engineering RNN reveals line attractor dynamics — Read a Paper</h3><p><a href="https://arxiv.org/abs/1906.10720">Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*B7kJOXi2L6U4KHtPGckbpQ.gif" /><figcaption>Source: [<a href="https://tenor.com/view/corona-virus-virus-queue-line-people-gif-16421639">Tenor GIF</a>]</figcaption></figure><p>With the recent trend of not stopping with performance analysis but rather extending it to understand the unfathomable interpretability and dynamics of the deep learning models, the paper talks about the learning dynamics of the Recurrent Neural Networks (RNNs) in dynamical system point of view. The observations put forth in the work are,</p><h4>Training reduces the dimensions of hidden unit activity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*5UgE5DLU_hrIrllgIYu-tw.jpeg" /><figcaption><a href="https://arxiv.org/abs/1906.10720">Source</a></figcaption></figure><p>RNN is trained with benchmark sentimental analysis dataset and the hidden unit activities of RNN is collected at each stage. At each stage PCA (Principal Component Analysis) is applied to check the required components to capture the activity. As training proceeded, the number of components of PCA required to explain variance reduced. This shows that, the training model indirectly impose parsimony but also manages to achieve generalization.</p><h4>State spaces for different classes physically separate</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*It0q5_5wTKPIfEWuJ8hdKw.jpeg" /><figcaption>2D histogram<br>Denoting the density of visited states when training in Yelp reviews dataset (negative -red and positive -green) <a href="https://arxiv.org/abs/1906.10720">Source</a></figcaption></figure><p>When training is started, the state vector for different class examples lies very close to each other, and as the training proceeds, they devolve further away. This shows that, the feature extraction to acquire distinction capability is also reflected in the geometry of the state space and that is clearly interpretable in visualization.</p><h4>Existence of Fixed points</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*K5gSIx2ev3lvaBzSkwUoIw.jpeg" /><figcaption>Projection of approximate fixed point onto Principal Components <a href="https://arxiv.org/abs/1906.10720">Source</a></figcaption></figure><p>During training, the RNN state vector follows a set of fixed points. In dynamical system point of view, a fixed point acts like a sink, that attracts the system to stay in that particular point. To kick the system out of the point, a sufficiently large force must be applied or the system will remain immutable or move away and rapidly fall back again. In deep learning perspective, the force is attributed to the saliency of the training example. When the data sample is neutral or has less salience (feeble positive or negative sentiment), the model in the fixed point is unaffected. But in the other hand, when a strong positive or negative review (example) is encountered, the model jumps out of the fixed point and settles in other fixed point. This also accounts for the robustness of the deep learning model against noise. And the manifold of all such fixed points lie one dimensional. That gives, another super cool conclusion that, the training model, not only settles in low dimensional state space but also forms 1d manifold of fixed points.</p><h4>Analysis of fixed point</h4><p>To find the approximate fixed points numerically, the paper defines a loss function, q = 1/N (Absolute Error(h — F(h, 0)).<strong> </strong>The function simply has a criterion to minimize the difference of hidden state activity with the one that is obtained when zero is given as input. When the hidden activity, is close to that of F(h,0), we can approximate that the point is fixed, and the function is optimized using optimization techniques. This again is consistent with the definition of fixed point, that is being immune to neutral (zero) and attracting the system quickly. Also, the authors verified whether the point is fixed by giving slight positive/negative example to perturb the system and checked the rate at which it falls back again.</p><h4>Stability of fixed point</h4><p>The authors also did stability analysis to check the nature of fixed point. Briefly stating, the first-order Taylor expansion is used to approximate hidden state update to get Jacobian. After eigen decomposition, left and right eigenvectors are obtained, to check the stability of the fixed point.</p><p>The experimentation is done across various RNN types, architectures and datasets to prove universality. The parsimony and linear approximation of learning dynamics of RNNs, draw many parallel analogies. One interesting parallel with Neuroscience is, the attractor dynamics that is believed to be observed in hippocampus [Ref 1] (known to play vital role in memory and learning), assumed to use simple low dimensional grid cells to form complex high dimensional representations[Ref 2].</p><p>Hope you enjoyed reading :)</p><h4>References</h4><ol><li><em>Tom J. Willsm Colin Lever, Francesca Cacucci, Neil Burgess, John O’Keefe (2005) </em><a href="https://science.sciencemag.org/content/308/5723/873.full"><em>Attractor Dynamics in the Hippocampal Representation of the Local Environment</em></a><em>. Science Vol. 308, Issue 5723, pp. 873–876</em></li><li><em>Klukas M, Lewis M, Fiete I (2020) </em><a href="https://pubmed.ncbi.nlm.nih.gov/32343687/"><em>Efficient and flexible representation of higher-dimensional cognitive variables with grid cells</em></a><em>. PLoS Comput Biol 16(4): e1007796.</em></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cae3e3f8638b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/read-a-paper/reverse-engineering-rnn-reveals-line-attractor-dynamics-read-a-paper-cae3e3f8638b">Reverse engineering RNN reveals line attractor dynamics — Read a Paper</a> was originally published in <a href="https://medium.com/read-a-paper">Read a Paper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>