<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dr. Tolga Ahmet Kalaycı on Medium]]></title>
        <description><![CDATA[Stories by Dr. Tolga Ahmet Kalaycı on Medium]]></description>
        <link>https://medium.com/@tolga68kalayci?source=rss-c56f374a8313------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fEkn0Iyu3yv41VY10z5l1A.jpeg</url>
            <title>Stories by Dr. Tolga Ahmet Kalaycı on Medium</title>
            <link>https://medium.com/@tolga68kalayci?source=rss-c56f374a8313------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:53:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tolga68kalayci/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Intuitions and Interpretations on Attention Mechanism in Deep Learning]]></title>
            <link>https://ai.plainenglish.io/intuitions-and-interpretations-on-attention-mechanism-in-deep-learning-b70ee23a9021?source=rss-c56f374a8313------2</link>
            <guid isPermaLink="false">https://medium.com/p/b70ee23a9021</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dr. Tolga Ahmet Kalaycı]]></dc:creator>
            <pubDate>Sun, 24 Sep 2023 07:44:46 GMT</pubDate>
            <atom:updated>2023-09-26T10:56:34.172Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What does attention mechanism do differently? Why it made such a difference in the era?</em></p><p>In this story, I would like to talk about some intuitive explanations on the attention mechanism. I will try to keep my words as non-technical as possible. My explanations will be based on text data however same reasonings and intuitions apply for image and other data types as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*RWE-BO7JeFbyNswdVkqu0A.jpeg" /><figcaption>source : <a href="https://www.evergreensgv.org/blog/2014/8/4/pay-attention">https://www.evergreensgv.org/blog/2014/8/4/pay-attention</a></figcaption></figure><p>Let’s begin with a brief overview on the basic working principles of deep neural networks.</p><p>The basic intention of popular deep learning architectures like CNN’s, RNN’s and Transformers is to discover the relations between the items that they are dealing with.</p><p>For example, a CNN that deals with an image tries to understand which pixels should be treated together to extract a meaningful information about that image.</p><p>Similarly, a RNN or a Transformer that is working on a sentence aims to understand which words ( <em>in fact “token” which means a “subword text” is a more proper expression instead of “word”, however I will not go in the details of this difference and keep using “word” for simplicity</em> ) relate to and / or affect the meaning of each other. In this way, for every single word in the sentence they come up with a group of relevant words each with a different relevancy weight.</p><p>This weighted group of words is called “<strong>context”. </strong>Obviously, since every word is numerically represented as vectors in deep learning, the context is also a numerical vector.</p><p>Let’s think about an English to French translation task such as below;</p><blockquote>On Medium, you have a huge audience and a great community → Sur Medium, vous avez une audience énorme et une grande communauté.</blockquote><p>Before the first introduction of attention mechanism by Bahdanau et al. [1] , the English sentence was being represented as a single context vector. Then, by using this context vector that basically carries information for the <strong>whole</strong> <strong>of the sentence</strong>, the French translation was being generated.</p><p>Do you notice a drawback here? Well, I do. Let’s think about the first two words of the sentence. <em>“On Medium,” </em>is translated as <em>“Sur Medium,”. </em>To make this translation do you really think that we need any information from the rest of the sentence?</p><p>Not at all, in fact caring about the rest of the sentence can even <strong>confuse</strong> us or we can <strong>forget</strong> about these two words while trying to memorize important details of the whole sentence.</p><p>You can argue that this is an exceptional example and that it is not encountered frequently, however <strong>this not the case</strong>. Just give yourself some time to try to analyze how your brain functions while making a translation.</p><p>I agree that it is not that straightforward to describe what is really going on in our brain for even such a short task. However I can argue that handling the English sentence part by part is more brain-like than memorizing all the details of the sentence at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*wTgEUxMztH79aSIeHLg5vQ.jpeg" /><figcaption>source: <a href="https://yourstory.com/2023/06/understanding-attention-mechanism-in-deep-learning">https://yourstory.com/2023/06/understanding-attention-mechanism-in-deep-learning</a></figcaption></figure><p>At this point, the first benefit of attention mechanism comes into the picture. Thanks to this mechanism, we no more need to limit the representation of the English sentence into a single context vector. On the contrary, to detect each French word we use a different context vector that gives more weight to (in other words “that attends to” — this is where the attentions mechanism gets its name) different parts of the English sentence.</p><p>The greater the sentence length, the greater the added value of the attention mechanism because as you would anticipate it is harder to memorize all the details of a long sentence at once.</p><p>Bahdanau et al. [1] introduced the attention mechanism to be used in conjunction with RNN’s. By design, RNN’ s have never been good at dealing with long sentences. Although the introduction of attention mechanism made RNN’s more robust against sentence length, a more fundamental change was needed to address this problem.</p><p>This change came with the study of Vaswani et al. [2] called as “Attention is all You Need”. In their study, they proposed a new network architecture that they call “Transformer”. In this architecture , they totally replaced the location biased context extraction technique of RNN’s with an approach that does not presume any linkage between the locational proximity and semantic relation of two words in a sentence.</p><p>Having said that the transformers replace the location biased context extraction technique of RNN’s, this does not mean that transformers are unaware of the position of a word within a sentence. They get this information via positional embeddings which are simply other vectors each representing a position in the sentence. Since positional embeddings are out of scope of this story, I do not go into this subject any further.</p><p>By design, RNN’ s think that if two words come one after the other then while creating the context vectors this formation <strong>must</strong> be payed attention.</p><blockquote><strong>In RNN point of view : “If two words come one after the other then while creating the context vectors this formation must be payed attention.”</strong></blockquote><p>The Transformer architecture changed this statement as follows; if two words come one after the other then while creating the context vectors this formation should be payed attention if necessary.</p><blockquote><strong>In Transformer point of view : “If two words come one after the other then while creating the context vectors this formation should be payed attention if necessary.”</strong></blockquote><p>So I believe the difference between these two statements above make the perspective change that came with the introduction of transformers pretty obvious. This architecture made us closer to build systems that are capable of mimicking our own way of thinking.</p><p>Transformers’ flexible design bases its context vector generation method solely on the variants of attention mechanism such as Dot Product attention, Bi-directional self attention and Masked self attention layers.</p><p>In my next story, I will talk about intuitions and interpretations on the role of these layers in transformer architecture together with a brief explanation of encoder and decoder concepts.</p><p>If you would like to hear more about this kind of contents that focus on discussing mathematical concepts in a non-mathematical and intuitive style, you can follow my profile, or add me on LinkedIn via the link below. Also, if you liked the story feel free to share it on your social media.</p><p><a href="https://www.linkedin.com/in/dr-tolga-ahmet-kalayci-61524314/">https://www.linkedin.com/in/dr-tolga-ahmet-kalayci-61524314/</a></p><p>Thanks for reading.</p><h3><strong>References</strong></h3><p>[1] <a href="https://arxiv.org/abs/1409.0473">https://arxiv.org/abs/1409.0473</a></p><p>[2] <a href="https://arxiv.org/abs/1706.03762">https://arxiv.org/abs/1706.03762</a></p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><strong><em>(X</em></strong>), <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/XxRS92b2"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b70ee23a9021" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/intuitions-and-interpretations-on-attention-mechanism-in-deep-learning-b70ee23a9021">Intuitions and Interpretations on Attention Mechanism in Deep Learning</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>