<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lucio Dery on Medium]]></title>
        <description><![CDATA[Stories by Lucio Dery on Medium]]></description>
        <link>https://medium.com/@luciodery?source=rss-c3c68470040------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*6YEsx6on1R4focee.</url>
            <title>Stories by Lucio Dery on Medium</title>
            <link>https://medium.com/@luciodery?source=rss-c3c68470040------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:26:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@luciodery/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[REVIEW OF :WHY DO DEEP AND CHEAP LEARNING WORK SO WELL ?]]></title>
            <link>https://medium.com/@luciodery/review-of-why-do-deep-and-cheap-learning-work-so-well-916ed5667ff?source=rss-c3c68470040------2</link>
            <guid isPermaLink="false">https://medium.com/p/916ed5667ff</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[hamiltonian]]></category>
            <category><![CDATA[physics]]></category>
            <dc:creator><![CDATA[Lucio Dery]]></dc:creator>
            <pubDate>Mon, 04 May 2020 02:43:26 GMT</pubDate>
            <atom:updated>2020-05-04T05:52:31.031Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>REVIEW OF : </strong><a href="https://arxiv.org/pdf/1608.08225.pdf"><strong>WHY DO DEEP AND CHEAP LEARNING WORK SO WELL ?</strong></a></h3><p><strong>An Introduction</strong></p><p>If you’re reading this blog post, I either eagerly shared it with you — this is my first post, yay — or, you chanced on <a href="https://nlp.stanford.edu/IR-book/html/htmledition/pagerank-1.html">it during a random walk around the internet</a>. No matter how, due to the waves that Deep Learning (DL) has made and continues to make in the field of Artificial Intelligence (A.I), understanding why and how they work is an important research area. This blog post seeks to provide, via a scattering of analogies, a friendly exploration of a Physics-inspired theory of why DL works so well.</p><p>Why is the rise of DL systems so puzzling to a lot of computer scientists, mathematicians and even Physicists ? DL systems or neural networks (NNs) are universal function approximators. What this means is that given any continuous function mapping inputs to outputs, under sufficient conditions of data and resources (neurons), we can find a NN that can approximate the function within a fixed margin of error (See this wonderful <a href="http://neuralnetworksanddeeplearning.com/chap4.html">visual proof </a>for further clarification) Now for the puzzle. Imagine you are walking through a farm with haystacks and you mistakenly drop a precious sewing needle. You know it’s somewhere in the farm, but you don’t know where. You know you can find the needle, but there is so much hay that you are not sure whether you can find it anytime soon — it might take an infinite amount of time to find it. This is the core of the mystery behind NNs. Even though they can approximate any function, there are infinitely many functions, and a lot of times, we are looking for a particular function (one that generalizes well to new data after seeing the training data) — a needle, in this haystack of possible functions. But somehow, finding this needle is not an impossible task for NNs — they do not take infinite time or infinite resources to conduct this search. This is the puzzle — how do NNs learn so efficiently ?</p><p>There have been many proposals about the possible mechanisms that NNs might be using to conduct this efficient search of the haystack. This post seeks to understand and discuss one such proposal from an unlikely source. However, before starting, let’s motivate why this problem is relevant. DL systems are being deployed in <a href="https://itnext.io/2017s-deep-learning-papers-on-investing-7489e8f59487">finance</a>, <a href="https://www.techemergence.com/machine-learning-healthcare-applications/">healthcare</a>, <a href="https://selfdrivingcars.mit.edu/">self driving cars</a> amongst others. As these systems become an essential part of our lives, it is increasingly important to make sure that they are safe and reliable — this is not possible without a reasonable understanding of why DL systems work. There have been several developments over the past decade that show the flaws of DL systems, see <a href="https://www.nytimes.com/2018/06/21/opinion/facial-analysis-technology-bias.html">this</a>, <a href="https://blog.openai.com/adversarial-example-research/">this</a> and <a href="http://www.dailymail.co.uk/sciencetech/article-5535969/Uber-self-driving-car-death-avoided.html">that</a>. One way to mitigate these blind spots is to have a deeper understanding of how and why NNs work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*G6Nj4GT-tPqzeYyUVmDTDw.jpeg" /></figure><p>In the paper “Why does deep and cheap learning work so well” by physicists, Max Tegmark and Henry Lin argue that the key to decoding the success of NN is in Physics and not in Mathematics as most people would think. Their paper makes an argument on two fronts. Why do we have cheap learning — the needle should be really hard to find but in practice it’s not that difficult. We need relatively few neurons to approximate functions that seem hard on the surface. Why do we need deep learning instead of shallow learning — why is one technique for searching for the needle so much more efficient than another even though both are guaranteed to find it eventually ?</p><p><strong>Why Cheap Learning</strong></p><p>Why is it possible for a cheap neural network (a small network made up of just a few neurons and a handful of layers) able to solve our needle in a haystack problem described above ?</p><p>As a baseline example, let us assume we are trying to solve the problem of determining whether an image is of a cat or something else. Thus, given x = pixels of image [AND] y = “cat” we are trying to find the probability distribution P(y | x) — the probability that this is a cat given the pixels of the image. The paper by Lin <em>et al</em> begin by drawing a strong connection between the hamiltonian in physics and the probability distribution P(y | x). Thus, finding P(y | x) reduces to finding a hamiltonian in some sense. But what is hamiltonian and why is this parallel important?</p><p>In Physics, a hamiltonian basically describes the total energy of a system. As you might imagine, not all functions can represent a physical hamiltonian — by “physical” I mean relating to a real world system. We do not move backwards in time, boxes on tables do not start moving by themselves; these observations allude to a constrained set of possibilities one might call physical. Since the probability distributions P(y | x) we care about in Machine Learning describe datasets gotten from the real world, governed by physical hamiltonians, we can begin to have some hope that the restrictions that apply to hamiltonians can be translated to restrictions on P(y|x) — our needle in the haystack . This assumption helps to reduce the size of the haystack in which we search for P(y | x). So what features of the hamiltonian help to constrain where in the barn we search for the elusive P ( y | x)?</p><ol><li>Low order polynomial : Physical hamiltonians are usually low order <a href="https://en.wikipedia.org/wiki/Polynomial#Definition">polynomials</a> with typical degree ranging from d = 2 to 4. This is because, usually in Physics we study systems perturbatively — meaning we try to understand what happens when we make a small change to a system. Higher polynomial orders of small changes are vanishingly smaller (x^10 of 10^–5 is insignificant in many modeling situations) and thus we can get away with discarding these terms. You can also think of the higher order terms as corresponding to lower, more microscopic levels of interaction and all we might be concerned about are the macroscopic interactions captured by lower order terms (see <a href="https://en.wikipedia.org/wiki/Renormalization_group">renormalization</a> group). Returning to our barn / haystack analogy, you can think of this constraint as giving us the assurance that we can search with our eyes instead of a microscope since our needle isn’t that small. But just because we know that P (y | x) takes this functional form doesn’t mean that NNs can achieve this form. In order to complete the picture, the authors show that a small NN — 4 hidden units + sigmoid activation — can approximate multiplication and thus, a NN can approximate any low order polynomial (since any polynomial can be constructed from combining adding and multiplication operations on atomic terms)</li><li>Locality : You affect your immediate surroundings, with your sphere of influence waning the farther out it is from you. This is also expected of hamiltonians and is called Locality in physics. Instead of searching the whole barn for the needle, a rational person would start by retracing their steps, beginning from where they are currently standing. The locality assumption allows us to rule out parts of the barn that we did not visit during or that are not close to us, making our search space narrower. Connecting it back to the assumption about polynomial forms of the hamiltonian, the authors point out that due to locality, some of the coefficients of the polynomial expansion will automatically be zero because the interaction terms only occur between inputs that are local to each other.</li><li>Symmetry : More often than not, physical hamiltonians exhibit symmetry. This means that under some changes/transformations, the hamiltonian does not change. Eg. The gravitational <a href="https://en.wikipedia.org/wiki/Potential_energy">potential energy</a> of a box does not change when the box is displaced horizontally. This symmetry means that there is a constraint on the possible form of the hamiltonian (gravitational potential energy in this case) for the box . In the example above, if the hamiltonian has a dependence on the horizontal displacement then it cannot be the right description of the potential energy of the box since that would mean changing the horizontal position changes the hamiltonian/energy. Symmetries allow us to narrow down the form of P( y | x ) even more. Whether or not there is a cat in the image does not depend on which day the image was taken or the background of the picture (most of the time). These symmetries allow us to rule out certain forms of P (y |x), further narrowing our search space. Connecting to the haystack analogy, this would be similar to having an imaginary barn with two identical halves, thus a needle dropped in one half will also appear in the other half. Thus we can get away with searching only one half of the barn.</li></ol><p>By drawing a parallel between physical hamiltonians and hamiltonians that govern the kind of distributions that we search for in DL problems, the authors constrain the set of functions that can be P(y |x ). We thus have to search a smaller section of the barn for our needle than we originally thought, thus making it more plausible that a few neurons in a neural network can approximate P(y|x)</p><p><strong>Why Deep Learning</strong></p><p>Deep neural networks have shown tremendous improvement over shallow/cheap networks in a lot of Artificial Intelligence tasks.[<a href="https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html">case of image classification</a>] : deeper is usually better. But why?</p><p>Intuitively you can rationalize this as follows. In the first section, we talked about how physics limits the space of functions we need to search through and how cheap learning can approximate these functions. Thus, if we have a cheap network that approximates P(y |x), at a minimum, adding more layers should theoretically not change our performance since these layers can approximate the identity function (just passing on the result from a lower layer to a higher layer) and give the same result as the cheap network. Thus, at a minimum, deep neural networks can learn what shallow networks can.</p><p>The authors present arguments to prove that deep networks should perform better than cheap networks and should in some cases, be more efficient at computing our desired P(y| x). The core of the author’s argument is the fact that the physical processes that give rise to the distributions that we seek to learn, P(y | x) are hierarchical. This is due to the fact that <a href="https://en.wikipedia.org/wiki/Classical_mechanics">classical mechanics</a> is fundamentally markovian, meaning, the generative processes for most of the data that we have follows a markovian structure where simple, low level features are passed through generative layers, each only depending on the previous layer’s output. Thus, the layers of deep neural networks are able to model each step in this hierarchy and to distill the information from lower layers (x — pixels) , to make conclusions about higher layers (y — cat or something else). They do this by a reversal of the generative process. Thus, after modeling this hierarchy and distilling the information required, deep neural networks, more easily than shallow ones can approximate our desired P(y | x) better.</p><p>Moving from the realm of accuracy to efficiency, the authors ask the question, does stacking more layers into our neural network necessarily mean that we are compromising efficiency : since we seemingly will be making more computations as we add more layers? In some cases, as they explain via their “no-flattening theorems”, stacking more layers can give us several computational advantages. The clearest example given was in the case of synaptic efficiency, as measured by the density of non-zero values in the weight matrices being learned by NNs. Sparse matrices (matrices with many zero entries) are more memory efficient and we have algorithms to perform multiplications more efficiently on these matrices. Thus given matrix <strong>C</strong> = <strong>A</strong>*<strong>B</strong>, explicitly representing the dense matrix C with 2 sparse matrices A and B can be more computationally efficient. This decomposition into 2 matrices parallels the addition of more layers to the NN.</p><p><strong>Limitations</strong></p><p>Even though I am in love with this paper and the compelling analysis and explanations it provides, there are several limitations or key points that I feel were not addressed.</p><p>The paper spends a lot of time narrowing down the space of functions that need to be searched. However, it does not address the issue of how to search in the first place. Just because we have narrowed down our search to a single room in the barn doesn’t mean we are guaranteed progress in finding the needle. We could spend an equally infinite amount of time searching if, for example, we forget parts of the room already searched. I believe that a holistic understanding of why NNs work would also involve understanding key aspects of search strategies — namely optimization algorithms and what priors these introduce to solving learning problems. The paper does address one form of network architecture — the hierarchical architectures like Convolutional NNs, but again, mostly in relation to the types of functions that are expected under physics.</p><p>Also, in the argument about why networks need to be deep, the authors make a <a href="https://en.wikipedia.org/wiki/Markov_property">markovian</a> assumption about the hierarchical generative processes that produce the datasets we typically see in Machine Learning. This assumption served as a foundation for much of their analysis. However, there are lots of problems resulting from generative processes that might have feedback, memory amongst other properties that the markovian assumption does not capture. I do not believe the success of RNNs can be explained within this framework.</p><p>REFERENCES</p><p>Lin, Henry W., Max Tegmark, and David Rolnick. “Why does deep and cheap learning work so well?.” <em>Journal of Statistical Physics</em> 168.6 (2017): 1223–1247</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=916ed5667ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>