<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Filippo Galli on Medium]]></title>
        <description><![CDATA[Stories by Filippo Galli on Medium]]></description>
        <link>https://medium.com/@filippo92galli?source=rss-6df8ae25f579------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ldjqcuaHBefWVBVo</url>
            <title>Stories by Filippo Galli on Medium</title>
            <link>https://medium.com/@filippo92galli?source=rss-6df8ae25f579------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:46:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@filippo92galli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hopfield Networks are useless. Here’s why you should learn them.]]></title>
            <link>https://medium.com/data-science/hopfield-networks-are-useless-heres-why-you-should-learn-them-f0930ebeadcd?source=rss-6df8ae25f579------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0930ebeadcd</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hopefield-network]]></category>
            <category><![CDATA[boltzmann-machines]]></category>
            <dc:creator><![CDATA[Filippo Galli]]></dc:creator>
            <pubDate>Tue, 16 Apr 2019 17:17:34 GMT</pubDate>
            <atom:updated>2021-06-14T12:34:44.391Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*75TsP3iRN7ec0-Ff" /><figcaption>Photo by <a href="https://unsplash.com/@ryanschroeder?utm_source=medium&amp;utm_medium=referral">Ryan Schroeder</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hopfield networks were invented in 1982 by<a href="https://www.pnas.org/content/79/8/2554"> J.J. Hopfield</a>, and by then a number of different neural network models have been put together giving way better performance and robustness in comparison. To my knowledge, they are mostly introduced and mentioned in textbooks when approaching Boltzmann Machines and Deep Belief Networks, since they are built upon Hopfield’s work.</p><p>Nevertheless, they provide such a different and alternative perspective on learning systems compared to the current state of deep learning that they are worth understanding, if anything, for the sake of it.</p><p>Let’s find out <em>how </em>they work.</p><h3><strong>A basic Hopfield Net and how it works</strong></h3><p>At its core a Hopfield Network is a model that can reconstruct data after being fed with corrupt versions of the same data.</p><p>We can describe it as a network of nodes — or units, or neurons — connected by links. Each unit has one of two states at any point in time, and we are going to assume these states can be +1 or -1. We can list the state of each unit at a given time in a vector V.</p><p>Links represent connections between units and they are symmetric. In other words, the link between node-i and node-j is identical whichever direction you go in the graph. In particular, what is identical is the number representing the strength of the connection between each unit. We can list these numbers in a matrix <em>W</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/1*fztpVL3Q2EpLzWg5gi6lEQ.jpeg" /><figcaption>Fig.1 — A Hopfield Network</figcaption></figure><p>If we have a network like the one in Fig. 1, we can describe it with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*n5VWSO7nmSM1x1Ltwzl-Kw.png" /><figcaption>A state vector and a weight matrix describe the graph at some point in time</figcaption></figure><p>How do we build <em>W</em>? We do it by first noticing that there is no connection between a node and itself, so we make this explicit by zeroing <em>waa, wbb, wcc, wdd</em>. We also know that links -or weights- are symmetric, so for instance <em>wab = wba</em>. So <em>W</em> is zero-diagonal and symmetric. The rest of the elements are filled in this way: <em>wab = Va · Vb</em>. The result is matrix <em>W</em>. This turns out to be a smart choice for the connection weights, as it follows the Hebbian rule we will see again later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*9WgANQ5c2e4R0ANvB9-dgA.png" /><figcaption>The weight matrix fully describes the network connections</figcaption></figure><p>Now that we built up the weight matrix we need to define a rule to determine the states of each node. Since we have binary states we can use this law:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*QEXsP9lVr7Vr9CX2NW_jMA.png" /><figcaption>State update rule</figcaption></figure><p>By using the weight matrix defined above we have, similarly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*Q5pR2dVWz6-7HiuIpTJm3Q.png" /></figure><p>Now we can get an intuition of how Hopfield Networks actually work.</p><p>Say we have a corrupt version of <em>V</em>, and we will call it <em>V’ = [1 -1 -1 -1]</em> , such that the last bit was reversed. We can initialize the network states with <em>V’</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/1*nJOCCLhDWmbmdZlQLGFNIQ.png" /><figcaption>Network initialized with corrupted states: d is now -1</figcaption></figure><p>We can proceed updating the network states with the rules we established before. For Va we will have:</p><p><em>Va = f(wab · Vb + wac · Vc + wad · Vd)</em> = +1</p><p>Note that <em>wab · Vb</em> and <em>wac · Vc</em> have both positive contributions to <em>x</em>, despite <em>Vb</em> and <em>Vc </em>being -1, pushing <em>Va</em> to be +1, while the corrupted bit contributes in the opposite direction. In other words, by building <em>W</em> as before, we force nodes to settle in opposite states when they are supposed to be of opposite sign, and to settle to the same state when they are supposed to be equal. Opposite states repel each other, while same states attract instead. This is one formulation of the Hebbian rule.</p><blockquote><em>Hebbian Rule: </em>Neurons that fire together, wire together</blockquote><p>Similarly:<br><em>Vb = f(wba ·Va + wbc · Vc + wbd · Vd) = </em>-1<em><br>Vc = f(wca · Va + wcb · Vb + wcd · Vd) = </em>-1<br>Now the corrupted bit:<br><em>Vd = f(wda · Va + wdc · Vc + wdb · Vb) =</em> +1</p><p>And <em>Va =</em> +1 attracts node <em>d</em> to be positive by having wda · Va&gt;0, while <em>Vc=Vd= </em>-1 repel node <em>d</em> by giving an overall positive contribution <em>wdc · Vc + wdb · Vd &gt;</em>0.</p><p>Nodes that were originally the same, are driven to be the same, nodes that were originally of opposite sign repel each other to be opposite.</p><p><strong>The original <em>V</em> is magically recovered!</strong></p><h3>But *<em>why*</em> does it work?</h3><p>So far we saw that once we completely define the network -its <em>W</em>- with a state vector <em>V</em> that we want to recover after corruption, we can do it by just updating the network states.</p><p>In other words, after initializing the network states with <em>V’ </em>we let the network evolve with the laws we defined before, and it will converge to the states we wanted in the first place. Not only that, but it will remain there no matter how many updates we keep on doing.</p><p>Let’s find out <em>why</em>.</p><p>We can define a function that depends on the states of the graph and the <em>W </em>matrix. We will call this function the Energy function associated with the network states and denote it with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*LItMRIv5NRZxeaNuC6e-oA.png" /><figcaption>Energy function in a Hopfield Network</figcaption></figure><h4>Result #1</h4><p>If a node <em>Vi</em> changes its state from +1 to -1 or vice versa, we will have that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*thcXe91mo-qt4GAJpE6czw.png" /></figure><blockquote>Now:<br>If <em>Vi</em> changed from -1 to +1, then <em>dVi </em>= +2<br>Which means <strong><em>x</em></strong><em> </em>has to be positive,<br>And in turn, the Energy delta has to be negative</blockquote><p>This means that if we update the network according to our rules, the energy function will always decrease, it is <em>monotonically decreasing</em>, and it will try to reach its lowest point.</p><h4><strong>Result #2</strong></h4><p>But is there a lowest point or will the energy keep decreasing to negative infinity?</p><p>In other words, we are trying to figure out if the Energy delta can be zero.<br>To have<em> dEi = 0</em> we need, for instance, to have <em>dVi = 0</em>, which is true only when <em>Vi(k-1)’ = Vi(k)’</em>, with<em> Vi(k-1)’</em> being the node state before the update and <em>Vi(k)’</em> after the update.</p><p>Let’s suppose we have <em>Vi(k-1)’ = +1</em>, we want <em>Vi(k)’ = +1</em>, or similarly <br><em>xi(k) &gt; 0</em>. <br>Which is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*JgtmAiPxQ8kyOt3ytcpJDQ.png" /></figure><p>But when <em>Vj(k-1)’ = Vj</em> then <em>xi(k) </em>is always positive!</p><p>In one shot, we showed that when the states assume the original value (the uncorrupted value) the Energy function will not change anymore. In other words, <em>dVi = 0</em> and the node will not update to different values — the configuration is said to be stable.</p><blockquote>So we proved the Energy is always decreasing until the network reaches a stable configuration of the node states. Even more, the stable configuration is the configuration that corresponds to the restored state vector, a local minimum of the energy function.</blockquote><p>There are a number of implementation details that were spared here, but a basic, working Hopfield Network is in this Jupyter Notebook I prepared <a href="https://github.com/philipjk/sklearn_stuff/blob/master/hopfield_networks.ipynb">here.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0930ebeadcd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/hopfield-networks-are-useless-heres-why-you-should-learn-them-f0930ebeadcd">Hopfield Networks are useless. Here’s why you should learn them.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>