<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arthur Pesah on Medium]]></title>
        <description><![CDATA[Stories by Arthur Pesah on Medium]]></description>
        <link>https://medium.com/@arthur.pesah?source=rss-c66e091abf91------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DbH8wWnQYSP0AtFVlfionQ.jpeg</url>
            <title>Stories by Arthur Pesah on Medium</title>
            <link>https://medium.com/@arthur.pesah?source=rss-c66e091abf91------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:27:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arthur.pesah/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Improve your scientific models with meta-learning and likelihood-free inference]]></title>
            <link>https://medium.com/data-science/improve-your-scientific-models-with-meta-learning-and-likelihood-free-inference-2f904d0bd7fa?source=rss-c66e091abf91------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f904d0bd7fa</guid>
            <category><![CDATA[meta-learning]]></category>
            <category><![CDATA[physics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Arthur Pesah]]></dc:creator>
            <pubDate>Mon, 17 Dec 2018 02:21:31 GMT</pubDate>
            <atom:updated>2021-07-21T07:24:30.060Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6YLZfzsOH7fynt_ub_p2lQ.jpeg" /><figcaption>CMS Particle Detector in the LHC accelerator at CERN. Comparison between reality and simulation (with Geant4)</figcaption></figure><h3>Improve your Scientific Models with Meta-Learning and Likelihood-Free Inference</h3><h4>Introduction to likelihood-free inference and distillation of the paper <a href="https://arxiv.org/abs/1811.12932">Recurrent Machines for Likelihood-Free Inference</a>, published at the <a href="http://metalearning.ml/">NeurIPS 2018 Workshop on Meta-Learning</a>.</h4><p><em>Article jointly written by Arthur Pesah and </em><a href="https://medium.com/u/5808df5ce3e6"><em>Antoine Wehenkel</em></a></p><h3>Motivation</h3><p>There are usually two ways of coming up with a new scientific theory:</p><ul><li>Starting from first principles, deducing the consequent laws, and coming up with experimental predictions in order to verify the theory</li><li>Starting from experiments and inferring the simplest laws that explain your data.</li></ul><p>The role of statistics and machine learning in science is usually related to the second kind of inference, also called <em>induction</em>.</p><p>Imagine for instance that you want to model the evolution of two populations (let’s say foxes and rabbits) in an environment. A simple model is the <a href="https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equations">Lotka-Volterra differential equation</a>: you consider the probability that an event such as “a fox eating a rabbit”, “a rabbit being born”, “a fox being born”, etc. happens in a small time interval, deduce a set of differential equations depending on those probabilities, and predict the evolution of the two animal populations by solving those equations. By comparing your prediction with the evolution of a real population, you can infer the best model parameters (probabilities) in this environment.</p><figure><img alt="Fox eating rabbit" src="https://cdn-images-1.medium.com/max/300/1*t9Lv2LZ6EJiutaVzKoQ3lQ.jpeg" /></figure><p>Modern theories require <strong>simulations</strong> in order to be linked to observations. It can be either a simple differential equation solver as in the case of the Lotka-Volterra model, or a complex Monte-Carlo simulator as they use in particle physics for instance.</p><p>By comparing the results of a simulation, i.e. the predictions of a model, with real data, it is then possible to know the correctness of your model and adjust it accordingly. If this process of going back and forth between the model and the experimental data is usually done manually, the question that any machine learning practitioner would ask is: can we do it automatically? Can we build a machine that takes a tweakable simulator and real data as input, and returns the version of the simulator that fits best some real data?</p><p>That’s the object of our recent work [1], where we trained a neural network to come up with the best sequence of simulator tweaks in order to approximate experimental data, capitalizing on the recent advances in the fields of likelihood-free inference and meta-learning.</p><h3>Likelihood-free inference</h3><p>Let’s rephrase our problem in a more formal way. We can model a simulator (also called generative model) by a stochastic function that takes some parameters <strong>θ</strong> and returns samples <strong>x</strong> drawn from a certain distribution (the so-called <em>model</em>).</p><p>This formalism applies to any scientific theory that includes randomness, as it’s very often the case in modern science (particle collisions are governed by the law of quantum physics which are intrinsically random, biological processes or chemical reactions often occur in a noisy environment, etc.).</p><p>Experimental data consist in a set of points living in the same space as the output of the simulator. The goal of inference is then to find the parameters <strong>θ </strong>such that the simulator generate points as close as possible to the real data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*328OmNFA4xBuj4xgLQdK9w.png" /></figure><p>Scientific fields using such simulators include:</p><ul><li>Population genetics. <strong>Model:</strong> <a href="https://en.wikipedia.org/wiki/Coalescent_theory">Coalescent theory</a>. <strong>Observation:</strong> the DNA of a current population. <strong>Parameters:</strong> the DNA of the common ancestor.</li><li>High-energy particle physics. <strong>Model</strong>: <a href="https://en.wikipedia.org/wiki/Standard_Model">Standard Model</a> of particle physics. <strong>Observation: </strong>output of the detector during a collision. <strong>Parameters: </strong>coupling constants of the Standard Model (like the mass of the particles or the strength of the different forces).</li><li>Computational neuroscience. <strong>Model: </strong><a href="https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model">Hodgkin-Huxley model</a>. <strong>Observation:</strong> evolution of the voltage of a neuron after activation. <strong>Parameters</strong>: biophysical parameters of the neuron.</li></ul><p>So how can we predict the parameters of a simulator given some real observations? Let’s consider the simple example of a Gaussian simulator, that takes a vector <strong>θ=</strong>(μ<strong>,</strong>σ) as parameters and returns samples from the Gaussian distribution 𝓝(μ<strong>,</strong>σ).</p><p>The classical way to infer the parameters of such a simulator is called <em>Maximum Likelihood Estimation (MLE)</em>. The likelihood of real data points X under a probability distribution P parametrized by <strong>θ </strong>(the simulator) is defined as P(X|<strong>θ</strong>). It means that if most data points are located in high density regions, the likelihood will be high. Hence, the best parameters of a model are often the ones that maximize the likelihood of real data. If you are unfamiliar with likelihood-based inference, you can read <a href="https://towardsdatascience.com/probability-concepts-explained-maximum-likelihood-estimation-c7b4342fdbb1">this excellent introduction to the subject</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*16t2IyuYfARkjea8mfUDTQ.png" /></figure><p>If you have explicitly access to the underlying probability distribution of your simulator, as well as the gradient of this distribution, you can for instance perform a gradient descent in the parameter space, in order to maximize the likelihood and infer the best parameters of your model.</p><p>However, many real-life simulators have an <strong>intractable likelihood</strong>, which means that the explicit probability distribution is too hard to compute (either analytically or numerically) . We must therefore find new ways to infer the optimal parameters without using neither the likelihood function nor its gradient.</p><p>To sum it up, we have a black-box stochastic simulator that takes parameters and generates samples from an unknown probability distribution, as well as real data that we are able to compare to the generated ones. Our goal is to find the parameters that lead the simulator to generate data as close as possible to the real ones. This setting is called <strong>likelihood-free inference</strong>.</p><h3>How can we likelihood-free infer?</h3><p>Let’s try to come up progressively with a method to solve our problem. The first thing we can do is to start from a random parameter, and simulate the corresponding data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Dgh8KZYJ_aAYYI73jy8wUQ.png" /><figcaption>Representation of the two spaces of interest. The true parameter (that we wish to infer) is the red point on the left. The real data correspond to the red cloud of points on the right. We start by choosing a random parameter θ (in gray on the left) and simulating the corresponding data points (in gray on the right)</figcaption></figure><p>By doing so, we can see how far our generated data are from the real data, but we have no way to know where to move in the parameter-space. Instead, let’s simulate several parameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*a6mfrolFz-s9CicMOULgvA.png" /></figure><p>To do so, we consider a distribution in the parameter-space, called <strong>proposal distribution</strong> and noted q(θ|ψ). If we choose q to be a Gaussian distribution, we will have ψ=(μ<strong>,</strong>σ). The first step consists in initializing ψ randomly. In the figure above, we considered ψ=μ for simplicity. Then, we can perform the following step until convergence:</p><ul><li>Sampling a few parameters from the proposal distribution: 4 parameters around ψ in the figure.</li><li>Generating data from them: the 4 clouds of points on the right.</li><li>Choosing a good direction to move ψ.</li></ul><p>The third step is the hard one. Intuitively, you would like to move ψ towards the orange and green parameters, since the corresponding predictions (orange and green cloud of points) are the closest to the real data. A set of methods, called <em>natural evolution strategies </em>[3], allow you to link the performance of each θ (in terms of similarity between its predictions and the real data) to a direction in the parameter space. A recent paper [4] use for instance the similarity measure given by a Generative Adversarial Network (GAN) to find the best direction. Even though those algorithms perform well in the general case, one can wonder if for a given simulator, it is not possible to find a better algorithm that would exploit the particular properties of this simulator. That’s where meta-learning comes into play!</p><h3>Meta-learning for optimization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*Wip2SwVt4aMqBPtE2Spffw.png" /></figure><p>The idea behind meta-learning is to learn how to learn, and in our case to <strong>learn the optimization process</strong>. The main idea, introduced in the paper <a href="https://arxiv.org/abs/1606.04474">Learning to learn by gradient descent by gradient descent</a> [2], is to use a recurrent neural network (RNN) to find the best descent direction at each iteration. Below is an example of sequence of points produced by a randomly initialized RNN:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*5Axutz0l_0TjxRupjEt-1Q.png" /></figure><p>Each descent direction is random and the last point produced is far from the minimum. During training, it should learn to exploit the gradient information at each point in order to move toward the minimum, giving something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*r2Ww8UZAmL3cBRds_p35OQ.png" /></figure><p>So how to train it? Generate many functions whose minimum you know, and ask the RNN to minimize the distance between the last point of the sequence and the real minimum.</p><h3>Learning to learn scientific models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*qKC9tH01bkQ_giLs_yGJxg.png" /></figure><p>In the case of likelihood-free inference, the RNN should return a sequence of proposal parameters ψ, given the real observations and the generated cloud of points at each step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*La-daJwmoy6ksOpAeIrFkQ.png" /></figure><p>Now, same question as for learning to learn by gradient descent, how do we train the RNN? Here, the trick is to generate many random parameters θ and to pretend for each one that it is the “true parameter”. We can then simulate each θ generated and obtain a set of “real observations”. The RNN can then be trained by passing it those real observations, looking at its final proposal distribution, and comparing it to the true parameter (that we know because we have generated it).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7j8kOuY8p0MlwLXnDVzOEg.png" /></figure><p>Let’s go through an example to clarify all of that. In the figures below, the proposal distribution is represented in color (red = high density). At the beginning, the RNN is initialized randomly and we evaluate it on our first generated true parameter θ (in red).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*_PA7u4qOMkAnVEMm73XPiQ.gif" /></figure><p>We can then backpropagate the loss into the RNN. After repeating this process for 200 different “true parameters”, this is what is should look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*j6eVXNAOHY3rQxD2AqjKWQ.gif" /></figure><p>We can see that it has learnt to exploit the information of the observation space to move towards the good parameter.</p><h3>Results</h3><p>We evaluated our model on different toy simulators, some where we the likelihood is known and some where it is not.</p><h4>Non-likelihood-free problem: Poisson Simulator</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/1*204e7qk6Ml3dmQhwbjx7rw.png" /><figcaption>Example of Poisson distributions for various parameters. Source: Wikipedia</figcaption></figure><p>The first simulator takes a parameter λ and draw samples from a Poisson distribution P(λ). The goal of this example was to see if we obtain comparable performance as the maximum likelihood estimator. Here are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*ALQZ7-7AD6RVjJqQpgXfMA.png" /><figcaption>Comparison of ALFI (Automatic Likelihood-Free Inference, the name of our model), to a maximum likelihood estimator (MLE). Those box-plots represent the distribution of the mean-squared errors between the true parameters and the expected value of the final proposal distributions.</figcaption></figure><p>We can see that the performance are comparable, even though we didn’t give our model access to the likelihood.</p><h4>Likelihood-free problem: Particle Physics Simulator</h4><p>To evaluate our model in a true likelihood-free setting, we considered a simplified particle physics model that simulate the collision of an electron and a positron turning into a muon and an antimuon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*_Zpy-kvc2wiMW-9KzowoLw.png" /><figcaption>Feynman diagram of the simulated process</figcaption></figure><p>The parameters of the simulator are the energy of the incoming particles and the Fermi constant, and the output is the angle between the two muons.</p><p>To evaluate our method, we compared the real observations with the ones generated by the last parameter found. Here are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gr7tU9hWoGt8FRkPpZKppQ.png" /><figcaption>Results of our method on a simple particle physics model. Comparison of the real observations (angles of the produced particles) with the ones generated by our predicted parameter.</figcaption></figure><h3>Discussion</h3><p>We saw what likelihood-free inference is and how meta-learning can be used to solve it by learning the best sequence of simulator tweaks to fit a model to the reality.</p><p>As most meta-learning models, a limitation is that it is hard to train. We had trouble scaling our method to more complex simulators, since meta-training requires a lot of simulator calls, which might be very slow in real-world settings. However, as the field of meta-learning makes progress, we hope that new methods will emerge to alleviate this problem and make it more scalable.</p><h3>References</h3><p>[1] A. Pesah, A. Wehenkel and G. Louppe, <a href="https://arxiv.org/abs/1811.12932">Recurrent Machines for Likelihood-Free Inference</a> (2018), NeurIPS 2018 Workshop on Meta-Learning</p><p>[2] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, N. de Freitas, <a href="https://arxiv.org/abs/1606.04474">Learning to learn gradient descent by gradient descent</a> (2016), NIPS 2016</p><p>[3] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peter, J. Schmidhuber, <a href="http://jmlr.org/papers/v15/wierstra14a.html">Natural Evolution Strategies</a> (2014), Journal of Machine Learning Research (JMLR).</p><p>[4] G. Louppe, J. Hermans, K. Cranmer, <a href="https://arxiv.org/abs/1707.07113">Adversarial Variational Optimization of Non-Differentiable Simulator</a> (2017), arXiv e-prints 1707.07113</p><h3>Image sources</h3><ul><li>Cover image adapted from <a href="https://www.lhc-france.fr/spip.php?article870">LHC France</a></li><li><a href="https://www.wallpaperflare.com/fox-eating-rabbit-vector-art-animals-minimalism-representation-wallpaper-qodee">Fox eating rabbit</a></li><li>All the other figures are original</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f904d0bd7fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/improve-your-scientific-models-with-meta-learning-and-likelihood-free-inference-2f904d0bd7fa">Improve your scientific models with meta-learning and likelihood-free inference</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recent Advances for a Better Understanding of Deep Learning]]></title>
            <link>https://medium.com/data-science/recent-advances-for-a-better-understanding-of-deep-learning-part-i-5ce34d1cc914?source=rss-c66e091abf91------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ce34d1cc914</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Arthur Pesah]]></dc:creator>
            <pubDate>Sun, 19 Aug 2018 13:23:32 GMT</pubDate>
            <atom:updated>2021-07-21T07:25:31.323Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SDQwUVc41CCu8EknNzo-Kg.jpeg" /></figure><blockquote>I would like to live in a world whose systems are build on <strong>rigorous, reliable, verifiable knowledge</strong>, and not on alchemy. […] Simple experiments and simple theorems are the <strong>building blocks</strong> that help understand complicated larger phenomena.</blockquote><p>This call for a better <strong>understanding</strong> of deep learning was the core of Ali Rahimi’s <a href="http://www.argmin.net/2017/12/05/kitchen-sinks/">Test-of-Time Award presentation</a> at NIPS in December 2017. By comparing deep learning with alchemy, the goal of Ali was not to dismiss the entire field, but <a href="http://www.argmin.net/2017/12/11/alchemy-addendum/">“to open a conversation”</a>. This goal <a href="https://syncedreview.com/2017/12/12/lecun-vs-rahimi-has-machine-learning-become-alchemy/">has definitely been achieved</a> and people <a href="https://twitter.com/RandomlyWalking/status/1017899452378550273">are still debating</a> whether our current practice of deep learning should be considered as alchemy, engineering or science.</p><p>Seven months later, the machine learning community gathered again, this time in Stockholm for the International Conference on Machine Learning (ICML). With more than 5,000 participants and 629 papers published, it was one of the most important events regarding fundamental machine learning research. And <strong>deep learning theory</strong> has become one of the biggest subjects of the conference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WPkxBI3eE3Dfr4ix9nSa2Q.jpeg" /></figure><p>This renewed interest was revealed on the first day, with one of the biggest rooms of the conference full of machine learning practitioners ready to listen to the tutorial Towards Theoretical Understanding of Deep Learning by Sanjeev Arora. In his talk, the Professor of computer science at Princeton summarized the current areas of deep learning theory research, by dividing them into four branches:</p><ul><li><strong>Non Convex Optimization</strong>: How can we understand the highly non-convex loss function associated with deep neural networks? Why does stochastic gradient descent even converge?</li><li><strong>Overparametrization and Generalization</strong>: In classical statistical theory, generalization depends on the number of parameters but not in deep learning. Why? Can we find another good measure of generalization?</li><li><strong>Role of Depth</strong>: How does depth help a neural network to converge? What is the link between depth and generalization?</li><li><strong>Generative Models</strong>: Why do Generative Adversarial Networks (GANs) work so well? What theoretical properties could we use to stabilize them or avoid mode collapse?</li></ul><p>In this series of articles, we will try to build intuition in those four areas based on the most recent papers, with a particular focus on ICML 2018.</p><p>This first article will focus on the mysteries of non-convex optimization for deep networks.</p><h3>Non-Convex Optimization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRCWfdXa7b-ak2nBtmwRvw.png" /></figure><blockquote>I bet a lot of you have tried training a deep net of your own from scratch and walked away feeling bad about yourself because you couldn’t get it to perform. I don’t think it’s your fault. I think it’s gradient descent’s fault.</blockquote><p>stated Ali Rahimi with a provocative tone in his talk at NIPS. Stochastic Gradient Descent (SGD) is indeed the cornerstone of deep learning. It is supposed to find a solution of a highly non-convex optimization problem, and understanding when it works or not, and why, is one the most fundamental questions we would have to adress in a general theory of deep learning. More specifically, the study of non-convex optimization for deep neural networks can be divided into two questions:</p><ul><li>What does the loss function look like?</li><li>Why does SGD converge?</li></ul><h3>What does the loss function look like?</h3><p>If I ask you to visualize a global minimum, it’s very likely that the first representation that will come to your mind will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*5oKwwQKZF-202vpj-21sLA.png" /></figure><p>And it’s normal. In a 2D-world, it’s not rare to find problems, where around a global minimum, your function will be <strong>strictly</strong> convex (which means that the two eigenvalues of the hessian matrix at this point will be both strictly positive). But in a world with billions of parameters, as it is the case in deep learning, what are the odds that none of the directions around a global minimum are flat? Or equivalently that the hessian contains not a single zero (or almost zero) eigenvalue?</p><p>One of the first comment of Sanjeev Arora in his tutorial was that the number of possible directions that you can take on a loss function grows exponentially with the dimension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*gJQ0qSmL4oALv053Pe58SA.png" /></figure><p>Then, intuitively, it seems likely that a global minimum will not be a point, but a <strong>connected manifold</strong>. Which means that if you’ve reached a global minimum, you should be able to walk around on a flat path where all the points are also minima. This has been experimentally proven on large networks by a team at Heidelberg University, in their paper<a href="https://icml.cc/Conferences/2018/Schedule?showEvent=2780"> Essentially No Barriers in Neural Network Energy Landscape</a> [1]. They argue an even more general statement, namely that any two global minima can be connected through a flat path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*T-TVaQ9NIVlBDFnfNE2-_Q.png" /></figure><p>It was already known to be the case for a CNN on MNIST or an RNN on PTB [2], but this work extended that knowledge to much bigger networks (some DenseNets and ResNets) trained on more advanced datasets (CIFAR10 and CIFAR100). To find this path, they used a heuristic coming from molecular statistical mechanics, called AutoNEB. The idea is to create an initial path (for instance linear) between your two minima, and to place pivots on that path. You then iteratively modify the positions of the pivots, such that it minimizes the loss of each pivot and make sure the distances between pivots stay about the same (by modelling the space between pivots by springs).</p><p>If they didn’t prove that result theoretically, they gave some intuitive explanations on why such path exists:</p><blockquote>If we perturb a single parameter, say by adding a small constant, but leave the others free to adapt to this change to still minimise the loss, it may be argued that by adjusting somewhat, the myriad other parameters can “make up” for the change imposed on only one of them</blockquote><p>Thus, the results of this paper can help us seeing minima in a different way, through the lens of overparametrization and high-dimensional spaces.</p><p>More generally, when thinking about the loss function of neural network, you should always have in mind that the number of possible directions at a given point is huge. Another consequence of that is the fact that saddle points must be much more abundant than local minima: at a given (critical) point, among the billions of possible directions, it’s very likely to find one that goes down (if you’re not in a global minimum). This intuition was formalized rigorously and proved empirically in a paper published at NIPS 2014: <a href="https://arxiv.org/abs/1406.2572">Identifying and attacking the saddle point problem in high-dimensional non-convex optimization</a> [6]</p><h3>Why does SGD converge (or not)?</h3><p>The second important question in optimization of deep neural networks is related to the convergence properties of SGD. While this algorithm has long been seen as a faster but approximate version of gradient descent, we now have evidence that SGD actually converges to better, more general, minima [3]. But can we formalize it and explain quantitatively the capacity of SGD to escape from local minima or saddle points?</p><h4>SGD modifies the loss function</h4><p>The paper <a href="https://arxiv.org/abs/1802.06175">An Alternative View: When Does SGD Escape Local Minima?</a> [4] showed that performing SGD is equivalent to doing regular gradient descent on a convolved (thus smoothed) loss function. With that point of view and under certain assumptions (shown by the authors to be often true in practice), they prove that SGD will manage to escape local minima and converge to a small region around a global minimum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-kqaijhX5oF_WOYlWpjD9Q.png" /></figure><h4>SGD is governed by stochastic differential equations</h4><p>Another approach to SGD that has really changed my vision of this algorithm is <em>continuous SGD</em>. The idea was presented by Yoshua Bengio during his talk <a href="http://www.iro.umontreal.ca/~bengioy/talks/ICMLW-nonconvex-14july2018.pptx.pdf">On stochastic gradient descent, flatness and generalization</a>, given at the ICML Workshop on Non-Convex Optimization. SGD does not move a point on a loss function, but a <strong>cloud of points</strong>, or in other words,<strong> a distribution</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eYoXBMpA_RaHh6JB_IGSTQ.png" /><figcaption>Slide extracted from the presentation <a href="http://www.iro.umontreal.ca/~bengioy/talks/ICMLW-nonconvex-14july2018.pptx.pdf">On stochastic gradient descent, flatness and generalization</a><em>, by Y. Bengio, at ICML 2018. He presented an alternative way to see SGD, where you replace points by distributions (clouds of points)</em></figcaption></figure><p>The size of this cloud of point (i.e. the variance of the associated distribution) is proportional to the factor <em>learning_rate / batch_size</em>. A proof of this is given in the amazing paper by Pratik Chaudhari and Stefano Soatto, <a href="https://arxiv.org/pdf/1710.11029.pdf">Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks</a> [5], that they presented during the Workshop on Geometry in Machine Learning. This formula is quite intuitive: a low batch size means a very noisy gradient (because computed on a very small subset of the dataset), and a high learning rate means noisy steps.</p><p>The consequence of seeing SGD as a distribution moving over time is that the equations governing the descent are now <a href="https://en.wikipedia.org/wiki/Stochastic_partial_differential_equation">stochastic partial differential equations</a>. More precisely, under certain assumptions, [5] showed that the governing equation is actually a <a href="https://en.wikipedia.org/wiki/Fokker%E2%80%93Planck_equation"><strong>Fokker-Planck equation</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Y-B6LilyeB-JH3GFrRGB-Q.jpeg" /><figcaption>Slide extracted from the presentation <em>High-dimensional Geometry and Dynamics of Stochastic Gradient Descent for Deep Networks, by P. Chaudhari and S. Soatto, at ICML 2018. They showed how to pass from a discrete system to a continuous one described by the Fokker-Plank equation</em></figcaption></figure><p>In statistical physics, this type of equations describes the evolution of particles exposed to a drag force (that drifts the distribution, i.e. moves its mean) and to random forces (that diffuse the distribution, i.e. increase its variance). In SGD, the drag force is modeled by the true gradient while the random forces correspond to noise inherent to the algorithm. As you can see in the slide above, the diffusion term is proportional to a temperature term <em>T=1/β=learning_rate/(2*batch_size)</em>, which shows once again the importance of this ratio!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/0*QpCzRwRpBTos22To.gif" /><figcaption>Evolution of a distribution under the Fokker-Planck equation. It drifts on the left and diffuses with time. Source: <a href="https://en.wikipedia.org/wiki/Fokker%E2%80%93Planck_equation">Wikipedia</a></figcaption></figure><p>Using this framework, Chaudhari and Soatto proved that our distribution will monotonically converge to a certain steady distribution (in the sense of the <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence">KL-divergence</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*zwJHas7Je9hGeixU2rcSpw.png" /><figcaption>One of the main theorems of [5], proving monotonic convergence of the distribution to a steady state (in the sense of the KL divergence). The second equation shows that minimizing F is equivalent to minimizing a certain potential ϕ as well as maximizing the entropy of the distribution (trade-off controlled by the temperature 1/β)</figcaption></figure><p>There are several interesting points to comment in the theorem above:</p><ul><li>The functional that is minimized by SGD can be rewritten as a sum of two terms (Eq. 11): the expectancy of a potential Φ, and the entropy of the distribution. The temperature <em>1/β </em>controls the trade-off between those two terms.</li><li>The potential Φ depends only on the data and the architecture of the network (and not the optimization process). If it is equal to the loss function, SGD will converge to a global minimum. However, the paper shows that it’s rarely the case, and knowing how far Φ is from the loss function will tell you how likely your SGD will converge.</li><li>The entropy of the final distribution depends on the ratio <em>learning_rate/batch_size</em> (the temperature). Intuitively, the entropy is related to the size of a distribution and having a high temperature often comes down to having a distribution with high variance, which usually means a flat minimum. Since flat minima are often considered to generalize better, it’s consistent with the empirical finding that high learning and low batch size often lead to better minima.</li></ul><p>Therefore, seeing SGD as a distribution moving over time showed us that <em>learning_rate/batch_size </em>is more meaningful than each hyperparameter separated regarding convergence and generalization. Moreover, it enabled the introduction of the potential of a network, related to convergence and that could give a good metric for architecture search.</p><h3>Conclusion</h3><p>The quest of finding a deep learning theory can be broken down into two parts: first, building intuitions on how and why it works, through toy models and experiments, then expressing those intuitions into a mathematical form that can help us explaining our current results and making new ones.</p><p>In this first article, we tried to convey more intuition of both the high-dimensional loss function of neural networks and the interpretations of SGD, while showing that new kinds of formalism are being built in the objective of having a real mathematical theory of deep neural networks optimization.</p><p>However, while non-convex optimization is the cornerstone of deep learning, its success comes mostly from its ability to generalize well despite a huge number of layers and parameters. This will be the object of the next part.</p><h3>References</h3><p>[1] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, Fred Hamprecht. <a href="https://icml.cc/Conferences/2018/Schedule?showEvent=2780">Essentially No Barriers in Neural Network Energy Landscape</a>, <em>ICML 2018.</em></p><p>[2] C. Daniel Freeman, Joan Bruna. <a href="https://arxiv.org/abs/1611.01540">Topology and Geometry of Half-Rectified Network Optimization</a>, a<em>rXiv:1611.01540</em>, 2016.</p><p>[3] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. <a href="https://arxiv.org/pdf/1609.04836.pdf">On large-batch training for deep learning: Generalization gap and sharp minima</a>, <em>ICLR 2017</em>.</p><p>[4] Robert Kleinberg, Yuanzhi Li, Yang Yuan. <a href="https://arxiv.org/abs/1802.06175">An Alternative View: When Does SGD Escape Local Minima?</a>, <em>ICML 2018</em></p><p>[5] Pratik Chaudhari, Stefano Soatto. <a href="https://arxiv.org/pdf/1710.11029.pdf">Stochastic gradient descent performs variational inference</a>, converges to limit cycles for deep network, <em>ICLR 2018</em></p><p>[6] Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio. <a href="https://arxiv.org/abs/1406.2572">Identifying and attacking the saddle point problem in high-dimensional non-convex optimization</a>, <em>NIPS 2014</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ce34d1cc914" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/recent-advances-for-a-better-understanding-of-deep-learning-part-i-5ce34d1cc914">Recent Advances for a Better Understanding of Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>