<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shaked Ben Guy on Medium]]></title>
        <description><![CDATA[Stories by Shaked Ben Guy on Medium]]></description>
        <link>https://medium.com/@shaked_52782?source=rss-9a005f09e998------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dx-BkiZzYzqxqUTLUxDWKQ@2x.jpeg</url>
            <title>Stories by Shaked Ben Guy on Medium</title>
            <link>https://medium.com/@shaked_52782?source=rss-9a005f09e998------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:31:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shaked_52782/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Graph Convolutional Neural Networks]]></title>
            <link>https://medium.com/@shaked_52782/graph-convolutional-neural-networks-71cc79c12b?source=rss-9a005f09e998------2</link>
            <guid isPermaLink="false">https://medium.com/p/71cc79c12b</guid>
            <dc:creator><![CDATA[Shaked Ben Guy]]></dc:creator>
            <pubDate>Thu, 31 Aug 2023 14:29:20 GMT</pubDate>
            <atom:updated>2023-09-13T15:11:42.407Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*r6Upn7y_u3mgpkzQNbZL-g@2x.jpeg" /></figure><p>In the last few years, various versions of Graph Neural Networks have been developed. Graph Convolutional Networks (GCNs) are one of these versions. GCNs are seen as a fundamental type of Graph Neural Network, and they have become an important part of this growing field.</p><p>In this article, we will explore the abstract concept of a graph, identify existing challenges in using graphs within deep networks, and examine how Graph Convolutional Networks (GCNs), developed by Thomas and Max Welling, address these problems. Additionally, we’ll delve into the theoretical and mathematical aspects of GCNs to gain a more profound understanding of the underlying mechanisms within Graph Convolutional Networks and how they function.</p><h4><strong>So what is a graph?</strong></h4><p>A graph is a visual representation of a collection of objects, wherein each pair of objects is connected to one another. Graphs are used to illustrate connections in various domains. A typical example is a social network, where each user can be symbolized by a code containing their information, and the edges denote the connections between users.</p><p>Mathematically, these interconnected entities are termed vertices or nodes, and the set of entities is denoted by V. The connections between the entities are referred to as edges, and the set of edges is denoted by E. The vertices can be represented as a subset of a cartesian product of the set of vertices: E⊆V×V, meaning that an edge is a pair of interconnected vertices. A graph composed of a set of vertices V and a set of edges E is denoted G(V,E).</p><p>In order to apply deep learning machinery to graphs, a neural network must be developed to process the graph data and extract its inherent characteristics and features. However, this task is not as simple as it seems, as the basic representation of a graph cannot be directly fed into a neural network as input. While we can try to utilize the edges and vertex tensors by incorporating them into the network, this approach falls short in providing the essential foundational information about the graph’s structure.</p><p>Hence, one can also integrate the adjacency matrix, which captures the relationships between the vertices’ positions. While this strategy may initially appear effective, it harbors two key limitations:</p><ol><li>There are graphs with many vertices but only a few edges. This leads to an adjacency matrix filled predominantly with zeros, making this approach computationally inefficient.</li><li>A single graph can have multiple adjacency matrices depending on the vertex arrangement. In other words, reordering the vertices of a random graph generates a new adjacency matrix. Consequently, feeding different matrices into the network could yield inconsistent outputs.</li></ol><p>Convolutional networks for graphs created a breakthrough in the field because they enabled processing and learning about graphs in a more efficient way.</p><h4><strong>So how does GCNN work?</strong></h4><p>Graph Convolutional Neural Networks (GCNNs) are a specialized form of deep neural networks optimized for working with graph data. Their main focus is on node classification, where they categorize each node in a graph based on its own features as well as those of its neighboring nodes.</p><p>In Graph Convolutional Neural Networks (GCNNs), each node in the graph is represented by a feature vector that captures its key characteristics.</p><p>The convolution operation blends the features of a node with those of its neighbors to update the node’s representation. The equation governing this propagation rule in the graph is as follows:</p><p><strong>Hⁱ⁺¹ = σ(ÂHⁱWⁱ)</strong></p><p>When <strong>H</strong> is the layer, <strong>W</strong> is the weight matrix, σ is the <strong>ReLU</strong> type activation function and <strong>Â</strong> is the connection of the credence matrix on the unit matrix <strong>(A + I)</strong> it is useful to add the connection of each node to itself in the credence matrix.</p><p>Managing the output range of the hidden layer is crucial to prevent issues like vanishing gradients. One strategy to address this is to normalize the adjacency matrix by multiplying it with the inverse of the diagonal node degree matrix, denoted as <strong>D̂⁻¹</strong>. The diagonal node degree matrix, <strong>D̂</strong> is a matrix where each diagonal entry. <strong>D̂ᵢᵢ</strong> signifies the degree of the corresponding node.</p><p>Consequently, the formula for the normalized output at each hidden layer can be expressed as follows:</p><p><strong>Hⁱ⁺¹ = σ(D̂⁻¹ÂHⁱWⁱ)</strong></p><p>Multiplying by the inverse of <strong>D</strong> effectively averages the values, akin to a spring-based mechanism. This normalization step helps in preserving the scale of the output feature vectors, ensuring that they neither become too large nor too small.</p><p>GCNNs usually have several convolutional layers. Each of these layers refines the node representations using the convolution process. By the time we reach the final layer, it provides the most refined and information-rich representations of the nodes.</p><p>To sum it up, GCNNs are advanced neural networks designed for graph analysis. They refine node representations through graph convolutions. As the network progresses through its multiple convolutional layers, these representations become increasingly precise, with the network weights optimized during training.</p><p>You are invited to enter Mike’s Twitter for additional and current content in the world of AI at the following <a href="https://x.com/mikee_3_14?s=21">link</a></p><p><strong>reference:</strong></p><ul><li><a href="https://towardsdatascience.com/understanding-graph-convolutional-networks-for-node-classification-a2bfdb7aba7b">Understanding Graph Convolutional Networks for Node Classification</a></li><li><a href="https://github.com/AvrahamRaviv/Deep-Learning-in-Hebrew">GitHub - AvrahamRaviv/Deep-Learning-in-Hebrew: ספר מלא בעברית על למידת מכונה ולמידה עמוקה</a></li><li><a href="https://www.data-to-viz.com/graph/network.html">Network diagram</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71cc79c12b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformers – Understanding The Architecture And How It Works]]></title>
            <link>https://medium.com/@shaked_52782/transformers-understand-the-architecture-and-how-it-works-ec324d25a17a?source=rss-9a005f09e998------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec324d25a17a</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Shaked Ben Guy]]></dc:creator>
            <pubDate>Thu, 20 Apr 2023 18:09:48 GMT</pubDate>
            <atom:updated>2023-06-12T21:53:57.172Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*b6yinv-Jt8-99OY3yurtZQ@2x.jpeg" /><figcaption>source: <a href="https://towardsdatascience.com/attention-is-all-you-need-discovering-the-transformer-paper-73e5ff5e0634">link</a></figcaption></figure><p>The Transformer architecture was published for the first time in the article <a href="https://arxiv.org/abs/1706.03762">&quot;Attention Is All You Need&quot;</a> [1] in 2017 and is currently a main pillar in the world of NLP and LLM in the field such as ChatGPT. But how does the Transformer actually work? In this article I will review all the parts of the Transformer by breaking down the architecture into parts and understanding each part of it.</p><p>to understand the complete architecture of the Transformer. We will first understand two main processes that take place in the Transformer:</p><ol><li><strong>Positional encoding</strong></li><li><strong>attention mechanism</strong></li></ol><p>After we understand the above processes we can easily connect them with the complete architecture of the Transformer and understand how it works.</p><p>let’s get started</p><h4>1. 𝗣𝗢𝗦𝗜𝗧𝗜𝗢𝗡𝗔𝗟 𝗘𝗡𝗖𝗢𝗗𝗜𝗡𝗚</h4><p>The placement and order of words are essential parts of all language. They define the grammar and thus the actual semantics of a sentence. RNNs naturally take word order as they analyze word by word sequentially.</p><p>But in the transformer architecture, this form of presentation does not exist and all the words enter the model at the same time, therefore one must take care of the way in which the position of the words in the sentence can be identified and this is where the position coding step comes into play.</p><p>to understand how to coding position works, I will use an example:</p><p>Suppose we want to arrange a sequence of integers in binary form, we can see that the lower a bit has a weight (LSB), the faster it changes. That is, the rate of change of the bit provides an indication of its position. In a similar way to the location coding operation, a vector P containing trigonometric functions sin(x) and cos(x) is created, thus creating a series whose members are functions with decreasing frequency, and according to the symbol that is added to each member of the series, an indication of its position can be obtained.</p><p>If we look at the pictures (1) and (2). &quot;pos&quot; will mark the position of a certain word in a sentence of any length, P0, P6 are defined to be the embedding position of a word in position 6 and 0 respectively, the length of the word will be marked by d and i to display each of the indexes. on the vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*M_PFxtXdb9vqb_SVSOipcw@2x.jpeg" /><figcaption>(1) Positions Embedding ; source: <a href="https://datascience.stackexchange.com/questions/51065/what-is-the-positional-encoding-in-the-transformer-model">link</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*8ffqI8krqmwELzqtX9itvQ@2x.jpeg" /><figcaption>(2) Positions Embedding ; source: <a href="https://datascience.stackexchange.com/questions/51065/what-is-the-positional-encoding-in-the-transformer-model">link</a></figcaption></figure><p>Let’s start with the picture (1). If we draw a classic graph of sin(x) and change pos to different values, we will get that indeed each vector P will be found in a different position on the graph in terms of the x axis, but due to the periodicity of sin(x) we encounter a problem in which it is possible that two vectors Pi and Pj. They will get the same value on the y-axis as you can see in the right picture.</p><p>The solution to this problem is the use of &lt;i&gt; values, which for even i values ​​uses sin(x) and for odd i values ​​uses cos(x).</p><p>In this example I will use sin(x) only the same for cos(x).</p><p>If we look at the picture (2), it seems that for each I get a function with a different frequency. And reading the value from the position coding formula against the different frequencies helps solve the periodicity problem and thus differentiate between position P0 and P6 in our example.</p><h4>2. 𝗔𝗧𝗧𝗘𝗡𝗧𝗜𝗢𝗡</h4><p>Now that we have understood and discussed the positional encoding stage and understood how it works, we can move on to the heart of the transformer, which is the attention mechanism.</p><p>The attention mechanism is the central mechanism on which the architecture of the transformer is based and it helps in finding a connection between organs.</p><p>In this post I will review in general how this mechanism works on a mathematical level.</p><p>To calculate the attention, create three matrices Query, Kry, Value. where each of them is created by multiplying a matrix of weights in the input element. With the help of these matrices, the attention is calculated according to the formula in picture (3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*na2Rwj2gx9wb09tyEioKqA@2x.jpeg" /><figcaption>(3) attention formula ; source: <a href="https://jalammar.github.io/illustrated-transformer/">link</a></figcaption></figure><p>To understand how this formula works and helps in finding a connection between organs, we will examine it in more detail. For input series X you get three matrices by multiplying the input series by the weights matrix. From here we get Q, K, V. Each element in the original series X forms a row in each of the matrices, when you take the matrix Q and multiply it in each of the rows in the matrix K, you get a new matrix (Q•K) that represents the correlation between each pair of elements in the original series. That is, each product q_i•k_j represents the relationship between member i and member j and indicates to what extent there is a relationship between members i,j in the original series. The result is divided by the root of the embedding dimension to keep the gradient stable, and then normalized by SoftMax. In this way, you get a matrix of numbers in the range [0,1], representing as mentioned the relationship between every two elements in the original series.</p><p>Now we will multiply the matrix we got by the matrix V and we will get a new representation for our data, Z, in which each vector z_i actually represents a member of the original series and includes information about the connections between it and the other members of the series.</p><h4>Full 𝗔𝗥𝗖𝗛𝗜𝗧𝗘𝗖𝗧𝗨𝗥𝗘</h4><p>Now we have reached the stage where all parts of the transformer can be connected together for the complete architecture of the model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*NNi9Dh0BuKUVs92V3_-s7w@2x.jpeg" /><figcaption>(4) Transformer architecture ; source : <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html">link</a></figcaption></figure><p>As you can see in picture (4), the transformer consists of two parts – encoder and decoder and includes the attention mechanism shown above three times for different purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*JNo7p6alOMvCHev9sR6KZA@2x.jpeg" /><figcaption>(5) Transformer – encoder ; source: <a href="https://builtin.com/artificial-intelligence/transformer-neural-network">link</a></figcaption></figure><p>The encoder (5) receives a certain series x after embedding and performs positional encoding on it. After that, the series goes through the residual block of self-attention, where the process I explained in the previous post is carried out. On this result we will add x in a similar way to the idea of ​​ResNet networks and we will get x+attention(x). Layer normalization is performed on this result. After that there is another residual block, which contains an FC layer and another normalization layer, and from there the encoder output goes to the decoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*19-S-MTok4qxoNFvR7BNmg@2x.jpeg" /><figcaption>(6) Transformer – decoder ; source: <a href="https://builtin.com/artificial-intelligence/transformer-neural-network">link</a></figcaption></figure><p>The decoder (6) is built in a very similar way, with two main differences:</p><p>its input is the output organs that existed until now, and in addition, at the beginning of the decoder there is a layer of Masked self-attention. This layer receives all the output members that have been there so far, and the purpose of the decoder is to learn by itself what the next member of the output is. This part is a bit confusing and the reason is more understandable towards the end of the article , so I will try to explain using an example that I will return to later:</p><p>suppose we received a series of labels for the output organs that include {cat, dog, shoe, person} we would like to “hide” these labels using some kind of mask and give our model “Guess” them.</p><p>In the first residual block, the decoder performs self-attention on the members of the series that have been received so far, and thus learns a new representation of them, which also contains the relationship between the members of this series, where also at the end the output x+attention(x) is received, which then undergoes layer normalization.</p><p>In the next step there is another multi-head attention layer encoder-decoder attention, where here the Key, Value and Query matrices come from the encoder and are passed as input to the decoder. Now when performing the Q•K multiplication, as you remember, you are not looking for similarity between the members of the same series but between the members of the output series of the encoder and the members of the input series in their representation after the masked layer.</p><p>That is, if we stop for a moment and summarize, we can say that in the encoder phase, the transformer looks for a connection between the members of the series and themselves in order to find the connections between the members of the series, but without yet knowing what the identified object is. And at the stage where the decoder compares the detected object with all the other possible objects, it is useful to know which object it is.</p><p>If we go back to our example for a moment, if we put x as input to the encoder of an image that includes the details we specified, first the model will learn the relationship between each pixel and the other pixels in the image. When pixels with a good correlation represent similar areas in the image, let’s say it is the man or any other object, and at the decoder stage an identification will be made between all the possible labels from the previous input in order to catalog that it is a man and not another object in the image.</p><p>This is actually the reason for performing the mask operation that I described earlier – if we don’t hide our labels, the model basically doesn’t learn anything.</p><p>The resulting representation will be passed through another residual block, which contains an FC layer and another normalization layer and at the end with another FC layer, and then SoftMax in order to analyze the data and classify it when in the end this image will be classified as a person.</p><p>References:</p><ul><li><a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a></li><li><a href="https://jyopari.github.io/Transformer.html">Transformer</a></li><li><a href="https://github.com/AvrahamRaviv/Deep-Learning-in-Hebrew">GitHub - AvrahamRaviv/Deep-Learning-in-Hebrew: ספר מלא בעברית על למידת מכונה ולמידה עמוקה</a></li><li><a href="https://datascience.stackexchange.com/questions/51065/what-is-the-positional-encoding-in-the-transformer-model">What is the positional encoding in the transformer model?</a></li><li><a href="https://jalammar.github.io/illustrated-transformer/">The Illustrated Transformer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec324d25a17a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>