<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marios Mourelatos on Medium]]></title>
        <description><![CDATA[Stories by Marios Mourelatos on Medium]]></description>
        <link>https://medium.com/@mourelatosm?source=rss-8d0f07c56a86------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*2vmNSzmzjg9tD8Is.</url>
            <title>Stories by Marios Mourelatos on Medium</title>
            <link>https://medium.com/@mourelatosm?source=rss-8d0f07c56a86------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:44:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mourelatosm/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Easy back propagation]]></title>
            <link>https://medium.com/@mourelatosm/easy-back-propagation-612b8a3a3fa1?source=rss-8d0f07c56a86------2</link>
            <guid isPermaLink="false">https://medium.com/p/612b8a3a3fa1</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[easy]]></category>
            <category><![CDATA[simple]]></category>
            <category><![CDATA[backpropagation]]></category>
            <dc:creator><![CDATA[Marios Mourelatos]]></dc:creator>
            <pubDate>Sun, 29 Mar 2020 21:14:00 GMT</pubDate>
            <atom:updated>2020-05-01T15:36:51.842Z</atom:updated>
            <content:encoded><![CDATA[<p>After many years of working with neural networks and automatic differentiation frameworks like PyTorch and TensorFlow I always found it quite annoying that the basic principle of those frameworks didn’t have a very short and easy to understand and remember explanation. On this article, I try to articulate with the simplest possible way how back propagation works.</p><p>Let’s start with the simplest possible neural network implementation in numpy. Numpy is a generic framework for scientific computing, it does not know anything about computation graphs and gradients. We can easily use numpy to fit a two layer network to random data by manually implementing the forward and backward passes:</p><pre><strong>import </strong>numpy <strong>as </strong>np<br><br><em># N is batch size; D_in is input dimension;<br># H is hidden dimension; D_out is output dimension.<br></em>N, D_in, H, D_out = 1, 2, 3, 1<br><br><em># Create random input and output data<br></em>x = np.random.randint(3, size=(N, D_in))<br>y_label = np.random.randint(3, size=(N, D_out))<br><br><em># Randomly initialise weights<br></em>w1 = np.random.randint(3, size=(D_in, H)).astype(float)<br>w2 = np.random.randint(3, size=(H, D_out)).astype(float)<br><br>learning_rate = 1e-4<br><strong>for </strong>t <strong>in </strong>range(5000):<br>    <em># Forward pass: compute predicted y<br>    </em>h = x.dot(w1)<br>    y = h.dot(w2)<br><br>    <em># Compute and print loss<br>    </em>loss = np.square(y - y_label).sum()<br>    print(t, loss)<br><br>    <em># Backprop to compute gradients of w1 and w2 with respect to loss<br>    </em>dL_dy = 2.0 * (y - y_label)<br>    dL_dw2 = h.T.dot(dL_dy)<br>    dy_dh = w2<br>    dL_dh = dL_dy.dot(dy_dh.T)<br>    dL_dw1 = x.T.dot(dL_dh)<br><br>    <em># Update weights<br>    </em>w1 -= learning_rate * dL_dw1<br>    w2 -= learning_rate * dL_dw2</pre><p>You can try the numpy script and find out that it learns quite easily. The part of the script that I want to focus on is how we compute the the gradients (or Deltas) we are going to update the weights with.</p><pre><em># Backprop to compute gradients of w1 and w2 with respect to loss<br></em>dL_dy = 2.0 * (y - y_label)<br>dL_dw2 = h.T.dot(dL_dy)<br>dy_dh = w2<br>dL_dh = dL_dy.dot(dy_dh.T)<br>dL_dw1 = x.T.dot(dL_dh)</pre><p>These 5 lines of code are applying the chain rule to compute the gradients of w1 and w2 with respect to loss and represent the foundation of the modern AI. Let’ s simulate a forward backward pass to build the intuition of how we wrote these 5 line of code.</p><p>As shown on the following diagram, after a forward pass from x to L all the values of the network like x, h, y and L are populated and we are ready to calculate the dL/dw1 and dL/dw2 accordingly. In order to find these derivatives it is important to define how many functions and how many variables we have on the graph. The rectangles represent variables and the ellipses functions. Each arrow represent a variable flow. For each arrow there is a corresponding partial derivative for the function of its destination with respect to the variable of its start. As a first step we can calculate all the partial derivates of the arrows of the graph. After this step we just need to apply the chain rule to calculate the desired dL/dw derivatives as shown on the green text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HknpgO28hZ3CCEAv3VGCLg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=612b8a3a3fa1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>