<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by José Miguel Hernández Lobato on Medium]]></title>
        <description><![CDATA[Stories by José Miguel Hernández Lobato on Medium]]></description>
        <link>https://medium.com/@jmhernandezlobato?source=rss-3d06198f9726------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vtkqsHSohoXA8kNI4LyVUA.png</url>
            <title>Stories by José Miguel Hernández Lobato on Medium</title>
            <link>https://medium.com/@jmhernandezlobato?source=rss-3d06198f9726------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:45:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jmhernandezlobato/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Successor Uncertainties]]></title>
            <link>https://medium.com/data-science/successor-uncertainties-b498097827fb?source=rss-3d06198f9726------2</link>
            <guid isPermaLink="false">https://medium.com/p/b498097827fb</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[José Miguel Hernández Lobato]]></dc:creator>
            <pubDate>Sun, 08 Dec 2019 09:06:06 GMT</pubDate>
            <atom:updated>2019-12-09T19:10:57.914Z</atom:updated>
            <content:encoded><![CDATA[<h4>Efficient Exploration in Model-free Reinforcement Learning</h4><p>I describe here our recent NeurIPS paper <a href="https://arxiv.org/abs/1810.06530">[1]</a> <a href="https://github.com/DavidJanz/successor_uncertainties_tabular">[code]</a>, which introduces <strong>Successor Uncertainties</strong> (SU), a state-of-the-art method for <strong>efficient exploration</strong> in <strong>model-free reinforcement learning.</strong></p><p>The leading authors are <strong>David Janz</strong> and <strong>Jiri Hron</strong>, two PhD students from the Cambridge <a href="http://mlg.eng.cam.ac.uk/"><strong>Machine Learning group</strong></a>, with the work having originated during an internship by David Janz at <a href="https://www.microsoft.com/en-us/research/lab/microsoft-research-cambridge/"><strong>Microsoft Research Cambridge</strong></a>.</p><p>The main insight behind SU is to describe Q-functions using probabilistic models that directly take into account <strong>correlations between Q-function values</strong>, as given by the <strong>Bellman equation</strong>. These correlations had been ignored in previous work.</p><p>The result is a method that <strong>outperforms competitors</strong> on tabular benchmarks and Atari games, while still being <strong>fast</strong> and <strong>highly scalable</strong>.</p><h3>Problem description</h3><p>We consider the general <strong>reinforcement learning</strong> (RL) setting in which an <strong>agent</strong> interacts with an <strong>environment</strong> with <strong>state</strong> s_t by taking <strong>action</strong> a_t. The environment then responds by transitioning to state s_t+1 and giving a reward value r_t+1, as shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*Rh44g4puAbLKnQlJGg9bJA.png" /></figure><p>The environment is usually <strong>stochastic </strong>and<strong> Markovian</strong>,<strong> </strong>fully determined by the unknown <strong>transition distribution</strong> P(s_t+1,r_t+1|s_t,a_t).</p><p>The agent selects its next action as a function of the current state using a <strong>policy function</strong> π(a|s), which is a probability distribution over actions given the current state, that is, π(a|s)=P(a_t=a|s_t=s).</p><p>The goal in RL is to find, from interactions with the environment, a policy function which maximizes the <strong>expected sum of discounted rewards</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*o3C-oSPpRtQtHvzaCLTOpA.png" /></figure><p>where 0≤γ&lt;1 guarantees the sum to be finite.</p><p>One of the <strong>main challenges </strong>in RL<strong> </strong>is that the effect of actions on rewards is often <strong>delayed</strong> and not noticeable until much later. This makes it difficult to identify the best action to take at any given state.</p><h3>Q-functions and generalized policy iteration</h3><p>When acting according to policy π, one way to account immediately about the effect of actions on future rewards is to use the <strong>state-action value function </strong>or<strong> Q-function.</strong></p><p>The Q-function returns the expected sum of discounted rewards that would be obtained when taking action “a” at state “s” and then acting according to π. In particular,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*2GmmBsM9A7WR4uY7FPvJrQ.png" /></figure><p>where the expectation is over the future actions chosen by π and the transition probabilities of the environment.</p><p>The Q-function can be shown to satisfy the <strong>Bellman equation</strong> since the value of an action at a particular state depends on the value of other actions at the <strong>successor states</strong>. In particular,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*CklFDrnMSuWeUNEAynSDUA.png" /></figure><p>This equation clearly shows that Q-functions are not arbitrary functions and <strong>must satisfy</strong> <strong>specific constraints</strong>. This is going to be <strong>key</strong> in the design of SU, which is the first method to account for these constraints when constructing probabilistic models of Q-functions.</p><h4>Policy improvement</h4><p>Q-functions are very useful because we can obtain a new policy π_new which is better or equal than π by <strong>acting greedily</strong> with respect to the Q-function for π. This process is called <strong>policy improvement</strong>. When the action space is discrete, we obtain the following greedy policy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*ipW7sm0ahu6O2qII3u85Ig.png" /></figure><h4>Policy evaluation</h4><p>To implement policy improvement, we need an estimate of the Q-function for the current policy. This estimate may not be initially available. One way to obtain such an estimate is by repeatedly applying the<strong> Bellman operator</strong> on an initial random guess of the Q-function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*wryp6RXzUlRDoc2b68nKiw.png" /></figure><p>which updates the Q-value for a state-action pair in terms of the Q-values of other state-action pairs. This process is called <strong>policy evaluation</strong>.</p><h4>Generalized policy iteration and efficient exploration</h4><p>Most model-free RL methods find optimal policies by starting with an initial random policy and then alternating policy evaluation and policy improvement steps with different degrees of granularity. This is called <strong>generalized policy iteration </strong>(GPI).</p><p>In many GPI methods, the expectations required by the Bellman operator in the policy evaluation step are often approximated by Monte Carlo, by <strong>averaging across data</strong> collected from interactions with the environment.</p><p>A key factor for good empirical results is then <strong>how to collect data</strong> so that convergence to an optimal policy is fast. <strong>Efficient exploration</strong> refers to the intelligent interaction with the environment that aims to make this fast convergence occur in practice.</p><p>However, most often, model-free RL methods do not perform efficient exploration and just collect data by following the greedy policy with probability 1-Ɛ and then choosing random actions with probability Ɛ.</p><h3>Thompson sampling and probabilistic Q-functions</h3><p>Thompson sampling (TS) is a <strong>strategy for efficient data collection</strong> which has shown to be <strong>highly successful across many different domains</strong>. In particular, our method, SU, uses an approximate version of TS for efficient exploration.</p><p>In the general setting, TS works by iteratively performing the following two steps:</p><ol><li>Sample hypothesis “H” about the data generating mechanism from a <strong>posterior distribution</strong> for “H” given the available data.</li><li>Collect new data by <strong>acting optimally</strong> according to the sampled “H”.</li></ol><p>The figure below shows an example iteration of TS when we want to maximize an unknown objective function.</p><p>Plot 1 shows data obtained by evaluating the objective at three different inputs. Plot 2 shows samples from a posterior distribution over objectives given the data collected so far. Plot 3 shows a selected sample and plot 4 shows the input that maximizes the sampled function. This input is the optimal action at which we would collect data next according to TS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTx0WEBvy8nsFsHgZhmu8A.png" /></figure><p>In the RL setting, TS is often called <strong>Posterior Sampling for Reinforcement Learning </strong>(PSRL) [5, 6]. When TS is applied to RL problems, “H” is a specific transition distribution and the posterior over “H” is computed given data in the form of observed tuples (r_t+1, s_t+1, s_t, a_t). Acting optimally with respect to a specific “H” involves then <strong>finding the optimal policy</strong> for the sampled P(s_t+1,r_t+1|s_t,a_t).</p><p>This means that step 2 above is <strong>highly costly</strong> in practice since, once we have sampled a specific transition distribution, acting optimally requires to solve a new RL problem specified by the sampled P(s_t+1,r_t+1|s_t,a_t).</p><p>The result is that <strong>Thompson sampling is infeasible</strong> in most RL problems.</p><h4>Probabilistic Q-functions</h4><p>The large computational cost of TS in RL has motivated researchers to come up with approximations.</p><p>A common approach is to work with a <strong>posterior distribution over Q-functions </strong>instead of with a posterior distribution over transition distributions. This approach is often called <strong>Randomized Value Function</strong> (RVF) [7]. In this case, acting optimally concerning a hypothesis sampled from the posterior distribution is <strong>as easy as computing the greedy policy</strong> for the sampled Q-function.</p><p>RVF has proven to be <strong>successful in practice</strong>, leading to RL methods that perform more <strong>efficient exploration</strong>. However, such methods <strong>still have limitations</strong> when Q-functions are approximated by neural networks.</p><p>First, obtaining a posterior distribution over Q-functions is not straightforward since, as we are performing off-policy RL, we do not have direct access to <strong>data in the form of inputs and corresponding output values </strong>for the Q-function. Many RVF methods using neural network approximations <strong>incorrectly assume that that is the case</strong>. Furthermore, these methods also have one of the following <strong>two problems</strong>:</p><ol><li>The distribution over Q-functions <strong>ignores dependencies between function values</strong> such as those given by the <strong>Bellman equation</strong>.</li><li>The distribution over Q-functions captures those dependencies, but at a <strong>large computational cost</strong>.</li></ol><p>SU does not have these problems since it captures dependencies given by the Bellman equation and has a low computational cost.</p><h3>Successor uncertainties</h3><p>A posterior distribution over Q-functions for a given policy π can be obtained by first, computing a posterior over transition distributions from observed tuples (r_t+1, s_t+1, s_t, a_t), and then mapping that posterior into a corresponding distribution over Q-functions.</p><p>Surprisingly, the previous operation can be done straightforwardly when the transition distributions satisfy the following assumptions:</p><ol><li>P(r_t+1|s_t,a_t) is given by the <strong>linear Gaussian model</strong> r_t+1 = <strong>w · ϕ</strong>(s_t,a_t)+ϵ_t, where <strong>ϕ</strong>(s_t,a_t) is a feature representation of states and actions and ϵ_t is Gaussian noise.</li><li>Uncertainty about P(s_t+1|s_t,a_t) is ignored.</li></ol><p>The previous linear model for the reward results in a corresponding <strong>linear model for the Q-function</strong>. In particular,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1_1MGmYfpBHEKk1TxJ1BsA.png" /></figure><p>where <strong>ψ</strong>(s_t,a_t) are the <strong>successor features </strong>[10], that is, the discounted expected future occurrence of each <strong>ϕ</strong>(s_t,a_t) under π.</p><p>Under a Gaussian prior for <strong>w</strong>, we obtain a Gaussian posterior N(<strong>w|m, S</strong>) for <strong>w </strong>which,<strong> </strong>under the previous equation, results into a <strong>fully correlated</strong> Gaussian posterior for the Q-function, with mean and covariance functions given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*Po5THFnXkhRkdWxRtSHVfg.png" /></figure><p>Since Q-functions are only defined for a given policy π, we assume this policy to be the <strong>average greedy policy</strong> obtained by sampling Q-functions from the posterior distribution.</p><p>This covariance function guarantees that the sampled Q-functions will satisfy <strong>the Bellman equation</strong>.</p><h4>Practical implementation</h4><p>The initial features <strong>ϕ</strong>(s_t,a_t) and the successor features <strong>ψ</strong>(s_t,a_t) are computed using a <strong>multi-head neural network</strong>, with one head per possible action value and feature type (<strong>ϕ </strong>or<strong> ψ)</strong>. The output of each head is multiplied by <strong>m</strong> to obtain predictions for the next reward and the Q-function value.</p><p>The whole process is illustrated in the figure below for the Atari game setting, with the input to the heads given by a convolutional neural network followed by a fully connected layer. In tabular environments, this part of the network can be replaced by a one-hot-encoding of the state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*hcPk_RambH2VL7kjrWn33w.png" /></figure><p>SU learns the neural network parameters and <strong>m </strong>by optimizing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*tQc4kPXkWJHp2f7b2OU5aQ.png" /></figure><p>in expectation over tuples (r_t+1, s_t+1, s_t, a_t) collected by interacting with the environment.</p><p>The objective above includes three different loss terms. The first one is the <strong>temporal difference error</strong> for the <strong>successor features</strong>. This term enforces <strong>ψ</strong> to converge to the discounted expected future occurrence of <strong>ϕ</strong>. The second loss term tunes <strong>m </strong>and <strong>ϕ </strong>so that the <strong>error in the reward predictions is low</strong>. Finally, the third term is the <strong>temporal difference error </strong>in the estimate of the <strong>Q-function</strong> and<strong> </strong>it<strong> </strong>enforces <strong>ψ</strong> to be also good for predicting the Q-function values.</p><p>Finally, the covariance matrix <strong>S</strong> is updated <strong>online according to Bayes rule in a Gaussian linear model</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*LaxzGwUtd4-vbQlk3hMDSQ.png" /></figure><p>where β is the noise variance in the linear model for the rewards and 0≤ζ≤1 is a decay factor that allows the model to forget, which is useful to counter the non-stationarity of <strong>ϕ </strong>during<strong> </strong>the learning process.<strong> </strong>In the experiments below we used β = 0.001 (binary tree) and β = 0.01 (Atari) and ζ = 1 (binary tree) and ζ = 0.99999 (Atari).</p><h3>Experiments on binary tree benchmark</h3><p>We evaluated the performance of SU on a challenging problem that requires <strong>efficient exploration</strong>. In this problem, see the figure below, there are <strong>2L+1</strong> states and two possible actions that are randomly mapped to movements <strong>UP</strong> and <strong>DOWN</strong> in each state when the environment is generated. The rewards are always 0 except after reaching state s_2L which produces a reward of 1. States with odd indices and s_2L are terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*zkqHQGCYkKLRDtYOOFoKuw.png" /></figure><p>This problem is <strong>challenging</strong> because an agent will always receive reward 0 except once the problem is solved, which, without intelligent exploration, will happen with an exponentially small probability as a function of <strong>L</strong>.</p><p>The figure below shows the median number of episodes that are required to learn the optimal policy. SU is compared with other RVF methods such as Bayesian DQN (<strong>BDQN</strong>) [3], the Uncertainty Bellman Equation (<strong>UBE</strong>) [2] and <strong>Bootstrapped DQN</strong> [4]. The latter method approximates the Q-function posterior with an ensemble of Q-function estimators.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8YEM1DFZo_HfExEmQIwcrQ.jpeg" /></figure><p>BDQN and UBE perform the same as <strong>a policy that samples actions uniformly</strong> <strong>at random</strong> and they both struggle to find optimal policies when <strong>L</strong> is large. Bootstrapped DQN is also worse than SU, with the differences becoming smaller when Bootstrapped DQN uses 25 times more computation than SU. The reason for this is that the analytic updates used by our Bayesian linear model allow for uncertainty to get accounted for really quickly, while bootstrap DQN relies more on gradient-based updates.</p><h3>Experiments on Atari games</h3><p>To show that it can be scaled to complex domains, we evaluated SU on the standard set of 49 Atari games [8]. Specific details of the implementation, network architecture and training procedure can be found in [1]. SU obtains a median human normalized score of 2.09 (averaged over 3 seeds) after 200M training frames under the ‘no-ops start 30 minute emulator time’ test protocol described in [9]. The table below shows that SU significantly outperforms competing methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*dVWaSYtJkVunADei1gQDHw.png" /></figure><p>For more details, the difference in human normalized score between SU and the competing algorithms for individual games is charted in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RJ5OLtgTt-i1zoFZnqdXOg.png" /><figcaption>Bars show the difference in human normalized score between SU and Bootstrap DQN (top),<br>UBE (middle) and DQN (bottom) for each of the 49 Atari 2600 games. Blue indicates SU performed<br>better, red worse. SU outperforms the baselines on 36/49, 43/49 and 42/49 games respectively.<br>Y-axis values have been clipped to [−2.5, 2.5].</figcaption></figure><h3>Summary</h3><p>Successor Uncertainties is a state-of-the-art method for <strong>efficient exploration</strong> in <strong>model-free reinforcement learning. </strong>In particular, SU</p><ol><li>is the first probabilistic method for Q-functions that incorporates <strong>dependencies given by the Bellman equation</strong>.</li><li>is <strong>highly scalable</strong> and as fast or faster than previous methods.</li><li><strong>outperforms competitors</strong> on tabular benchmarks and Atari games.</li></ol><h3>References</h3><p>[1] Janz* D., Hron* J., Mazur P., Hofmann K., Hernández-Lobato J. M. and Tschiatschek S. <strong>Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning</strong>, In NeurIPS, 2019. * Equal contributors.</p><p>[2] O’Donoghue, B., Osband, I., Munos, R., and Mnih, V. <strong>The uncertainty Bellman equation and exploration</strong>. In ICML, 2018.</p><p>[3] Azizzadenesheli, K., Brunskill, E., and Anandkumar, A. <strong>Efficient exploration through Bayesian deep Q-networks</strong>. In ICLR, 2018. <a href="https://openreview.net/forum?id=Bk6qQGWRb">https://openreview.net/forum?id=Bk6qQGWRb</a></p><p>[4] Osband, I., Blundell, C., Pritzel, A., &amp; Van Roy, B. <strong>Deep exploration via bootstrapped DQN</strong>. In NeurIPS, 2016.</p><p>[5] Strens, M. <strong>A Bayesian framework for reinforcement learning</strong>. In ICML, 2000.</p><p>[6] Osband, I., Russo, D., and Van Roy, B. <strong>(More) efficient reinforcement learning via posterior sampling</strong>. In NeurIPS, 2013.</p><p>[7] Osband, I., Van Roy, B., and Wen, Z. <strong>Generalization and exploration via randomized value functions</strong>. In ICML, 2016.</p><p>[8] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. <strong>Human-level control through deep reinforcement learning</strong>. Nature, 518(7540):529, 2015.</p><p>[9] Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M. G., and Silver, D. <strong>Rainbow: combining improvements in deep reinforcement learning</strong>. In AAAI Conference on Artificial Intelligence, 2018.</p><p>[10] Dayan, P. <strong>Improving generalization for temporal difference learning: the successor representation</strong>. Neural Computation, 5(4):613–624, 1993.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b498097827fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/successor-uncertainties-b498097827fb">Successor Uncertainties</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayesian Neural Networks with Random Inputs for Model Based Reinforcement Learning]]></title>
            <link>https://medium.com/data-science/bayesian-neural-networks-with-random-inputs-for-model-based-reinforcement-learning-36606a9399b4?source=rss-3d06198f9726------2</link>
            <guid isPermaLink="false">https://medium.com/p/36606a9399b4</guid>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[José Miguel Hernández Lobato]]></dc:creator>
            <pubDate>Fri, 21 Jul 2017 21:56:12 GMT</pubDate>
            <atom:updated>2017-10-04T20:41:29.735Z</atom:updated>
            <content:encoded><![CDATA[<p>I describe here our recent ICLR paper [1] [<a href="https://github.com/siemens/policy_search_bb-alpha">code</a>] [<a href="https://www.youtube.com/watch?v=0H3EkUPENSY)">talk</a>], which introduces a novel method for model-based reinforcement learning. The main author of this work is <strong>Stefan Depeweg</strong>, a phd student at Technical University in Munich who I am co-supervising.</p><p>The key contribution is in our models: <strong>Bayesian neural networks with random inputs</strong>, whose input layer contains both input features but also random variables which are propagated forward through the network and transformed into an arbitrary noise signal at the output layer.</p><p>The random inputs enable our models to <strong>automatically</strong> capture <strong>complex noise patterns</strong>, improving the quality of our model-based simulations and producing better policies in practice.</p><h3><strong>Problem description</strong></h3><p>We address the problem of policy search in stochastic dynamical systems. For example, to operate industrial systems such as gas turbines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*U7GN3vcDfOuVgLZtAlq8wg.png" /></figure><p>An abstraction of these systems is shown below. The current state of the system is denoted as s_t and associated to each state s_t there is a cost <em>c</em>(s_t) given by the function <em>c</em>( · ). At each time step we apply an action a_t that will affect the state of the system at the next time step s_t+1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/0*q0tODjpVTLSzreiS." /></figure><p>The transition from s_t to s_t+1 is not determined only by the action a_t, but also by some noise signal which we cannot control. This noise signal is represented by the dice in the figure. In the turbine example noise originates because the state that we observe consists only of sensor measurements which are an incomplete description of the true state of the system.</p><p>To control the system, we can use a policy function a_t = 𝜋(s_t;𝜃) to map the current state s_t into an action a_t. For example, 𝜋( · ;𝜃) could be a neural network with weights 𝜃.</p><p>Our goal is to find a policy (a value of 𝜃) that will produce <strong>on average</strong> low values of the cost function over sequences of state trajectories. For example, we aim to minimize the expectation of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/0*gxObBzlVunpFfIMg.png" /></figure><p>Note that the above expression is random because it depends on the choice of the initial state s_1 and on the random noise in the state transitions.</p><h3><strong>Batch reinforcement learning</strong></h3><p>We consider the <strong>batch reinforcement learning scenario</strong> in which we will not interact with the system during learning. This scenario is common in real-world industrial settings such as turbine control, where exploration is restricted to avoid possible damage to the system.</p><p>Therefore, to find an optimal policy we only have available a batch of data <em>D</em> = {(s_t, a_t, s_t+1)} in the form of state transitions obtained from an already running system and we will not be able to collect any additional data.</p><p>First, we learn from <em>D</em> a model for p(s_t+1|s_t, a_t), that is, the predictive distribution for the next state s_t+1 as a function of the current state s_t and the action a_t applied. We then connect this model with the policy to obtain p(s_t+1|s_t, a_t = 𝜋(s_t;𝜃)), which describes the evolution of the system when controlled with the policy 𝜋( · ;𝜃).</p><p>The previous distribution can be used to perform <strong>roll-outs</strong> or simulations of state trajectories. We start with a randomly sampled state s_1 and then iteratively sample from p(s_t+1|s_t, a_t = 𝜋(s_t;𝜃)) to obtain a trajectory of states s_1,…,s_T.</p><p>The cost function can then be evaluated on the sampled s_1,…,s_T to approximate cost(𝜃). The gradients of this approximation can be used to perform <strong>stochastic optimization</strong> of and move in directions that will produce on average low values of cost(𝜃).</p><h3><strong>The effect of noise in optimal control</strong></h3><p>The optimal policy can be significantly affected by the noise present in the state transitions. This is illustrated by the <strong>drunken spider</strong> story, which was originally proposed by Bert Kappen [2] and we use here as a motivating example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*xwuUeYgaIZuJVQyYKR34Jg.png" /></figure><p>A spider has two possible paths to go home: either by crossing the bridge or by walking around the lake. In the absence of noise, the bridge option is prefered since it is shorter. However, after heavily drinking alcohol, the spider’s movements may randomly deviate left or right. Since the bridge is narrow, and spiders do not like swimming, the prefered trajectory is now to walk around the lake.</p><p>The previous example shows how <strong>noise can significantly affect optimal control</strong>. For example, the optimal policy may change depending on whether the level of noise is high or low. Therefore, we expect to obtain significant improvements in model-based reinforcement learning by capturing with high accuracy any noise patterns present in the state transition data.</p><h3><strong>Bayesian neural networks with random inputs</strong></h3><p>In practice most modeling approaches for state transition data just assume <strong>additive Gaussian noise</strong> in s_t+1, namely,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/0*w8b8h-T0jahs84qL.png" /></figure><p>where <em>f_W</em> is for example a neural network with weights <em>W</em>. In this case it is very easy to learn <em>W</em> by maximum likelihood. However, the assumption of additive Gaussian noise is unlikely to hold in real-world settings.</p><p>A more flexible model for the noise in the transition dynamics can be obtained by using <strong>random inputs</strong> in <em>f_W</em>. In particular, we can assume that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/0*6cweHELT2GpWKIzw.png" /></figure><p>Under this model, the input noise variables z_t can be transformed in complicated ways through <em>f_W</em> to produce arbitrary random patterns in s_t+1 as a function of s_t and a_t.</p><p>However, now learning <em>W</em> can no longer be done by maximum likelihood because the z_t are unknown. A solution is instead to follow a <strong>Bayesian approach</strong> and work with the posterior distribution over <em>W</em> and the z_t. This distribution captures our uncertainty about the possible values that these variables may have taken after we see the data in <em>D</em>.</p><p>Computing the exact posterior is intractable, but we can learn a <strong>Gaussian approximation</strong>. The parameters of this approximation can be adjusted by minimizing a divergence with respect to the true posterior.<strong> Variational Bayes (VB)</strong> is a popular method for this that works by minimizing the Kullback-Leibler divergence.</p><h3><strong>α-divergence minimization</strong></h3><p>Instead of using VB, we learn our factorized Gaussian approximation <em>q</em> by <strong>minimizing </strong>the<strong> α-divergence </strong>[3,4]<strong>. </strong>By changing the value of α in this divergence, we can smoothly interpolate between solutions that fit a mode in the true posterior <em>p</em> or that aim to cover multiple modes in <em>p</em>, as illustrated in the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/0*_o4rUl-yxs8YTBpH." /></figure><p>Interestingly, VB is a particular case of α-divergence minimization with α = 0. Another well-known method for approximate Bayesian inference is expectation propagation, which is obtained with α = 1. In our experiments, we use <strong>α = 0.5</strong> since this often produces better probabilistic predictions in practice [4].</p><h3><strong>Results on a toy example</strong></h3><p>The figure below illustrates the results of our Bayesian neural networks with random inputs in two toy examples. The training data for each example is shown in the leftmost column. The top row shows a problem with a <strong>bi-modal</strong> predictive distribution. The bottom row shows a problem with <strong>heteroskedastic</strong> noise (the noise magnitude is input dependent).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H-qLdWDICOC_iljd." /></figure><p>The middle column shows the predictions obtained with a model that assumes only additive Gaussian noise. This model is unable to capture the bi-modality or the heteroskedasticity in the data. The rightmost column shows the predictions of our Bayesian neural networks with random inputs, which can automatically identify the type of random patterns present in the data.</p><h3><strong>Results on wetchicken problem</strong></h3><p>We consider now a reinforcement learning benchmark in which a canoeist is paddling on a two-dimensional river, as shown in the leftmost plot in the figure below. There is a <strong>drift</strong> in the river that pushes the canoeist towards a <strong>waterfall</strong> located in the top, with the drift being stronger on the right and weaker on the left. If the canoeist falls through the waterfall he has to start again at the bottom of the river.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7Kw4V6VCSCXNahp6." /></figure><p>There are also <strong>turbulences</strong> in the river which become stronger on the left and weaker on the right. The canoeist receives higher reward the closer he is to the waterfall. Therefore, he will want to be close to the waterfall, but not too close so that he may fall through. This problem is called <em>wetchicken</em> because of its similarities with the game <em>chicken</em>.</p><p>The turbulence and the waterfall will make wetchicken a highly stochastic benchmark: the possibility of falling down the waterfall induces a bi-modality in the state transitions, whereas the varying turbulence introduces heteroskedasticity.</p><p>The plot in the middle of the figure visualizes the policy found using our Bayesian neural networks with random inputs. This is a nearly optimal policy in which the canoeist paddles to try to stay at the location x ≃ 3.5 and y ≃ 2.5.</p><p>The plot in the right shows the policy found with a Gaussian process (GP) model that just assumes additive Gaussian noise. The resulting policy performs very poorly in practice because the GP cannot capture the complex noise patterns present in the data.</p><h3><strong>Results on the industrial benchmark</strong></h3><p>We also evaluate the performance of our Bayesian neural networks with random inputs in experiments with a simulator of industrial systems called the <em>“industrial benchmark” </em>[5]. According to the authors: “The “industrial benchmark” aims at being realistic in the sense, that it includes a variety of aspects that we found to be vital in industrial applications.”</p><p>The figure below shows, for a fixed sequence of actions, the roll-outs produced with models corresponding to 1) a multilayer perceptron which assumes additive Gaussian noise (MLP) and our Bayesian neural networks trained with 2) Variational Bayes (VB) or 3) α-divergence minimization with α = 0.5. Simulated trajectories are shown in blue and the ground truth one generated by the “industrial benchmark” is shown in red.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LNHJBgD4rs6M_1BR." /></figure><p>This figure clearly shows how the roll-outs produced with our Bayesian neural networks with random inputs and α-divergence minimization are closer to the ground truth trajectory.</p><h3><strong>Conclusions</strong></h3><p>We have seen that it is important to account for complicated noise patterns in the transition dynamics when learning optimal policies. Our Bayesian neural networks with random inputs are state-of-the-art models for capturing such complex noise patterns. By minimizing α-divergences with<strong> </strong>α = 0.5, we are able to perform accurate approximate inference in such Bayesian neural networks. This allows us to produce realistic model-based simulations that can be used to learn better policies.</p><h3><strong>Further reading</strong></h3><p>In [6] we study the decomposition of uncertainty in the predictions of Bayesian neural networks with random inputs. Uncertainty originates from either a) the lack of knowledge about the network weights due to limited data (epistemic uncertainty), or b) the random inputs to the network (aleatoric uncertainty). In [6] we show how to separate these two types of uncertainty with applications to active learning and safe reinforcement learning.</p><p>We also recommend <a href="http://alexgkendall.com/computer_vision/bayesian_deep_learning_for_safe_ai/">this</a><strong> </strong>excellent blog post by Alex Kendall about the aforementioned two types of uncertainty in deep neural network for computer vision.</p><h3>References</h3><p>[1] Depeweg S., Hernández-Lobato J. M., Doshi-Velez F. and Udluft S. <strong>Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks</strong>, In ICLR, 2017.</p><p>[2] H.J. Kappen. <strong>Path integrals and symmetry breaking for optimal control theory.</strong> Journal of statistical mechanics: theory and Experiment, page P11011, 2005.</p><p>[3] Minka, Thomas P. <strong>Divergence measures and message passing</strong>. Technical report, Microsoft Research, 2005.</p><p>[4] Hernández-Lobato J. M., Li Y., Rowland M., Bui T. D., Hernández-Lobato D. and Turner R. E.<strong> Black-Box Alpha Divergence Minimization</strong>, In ICML, 2016</p><p>[5] Daniel Hein, Alexander Hentschel, Volkmar Sterzing, Michel Tokic, and Steffen Udluft. <strong>Introduction to the” industrial benchmark”</strong>. arXiv preprint arXiv:1610.03793, 2016</p><p>[6] Depeweg, Stefan, et al. “<strong>Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables</strong>.” <em>arXiv preprint arXiv:1706.08495</em> (2017).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36606a9399b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/bayesian-neural-networks-with-random-inputs-for-model-based-reinforcement-learning-36606a9399b4">Bayesian Neural Networks with Random Inputs for Model Based Reinforcement Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural networks with optimal accuracy and speed in their predictions]]></title>
            <link>https://medium.com/data-science/neural-networks-with-optimal-accuracy-and-speed-in-their-predictions-d2cdc3b21b50?source=rss-3d06198f9726------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2cdc3b21b50</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[José Miguel Hernández Lobato]]></dc:creator>
            <pubDate>Thu, 20 Jul 2017 08:54:26 GMT</pubDate>
            <atom:updated>2017-07-23T09:06:29.578Z</atom:updated>
            <content:encoded><![CDATA[<p>Deep neural networks are currently used by companies such as Facebook, Google, Apple, etc. for making predictions from massive amounts of user generated data. A few examples are the Deep Face and Deep Text systems used by Facebook for face recognition and text understanding or the speech recognition systems used by Siri and Google Now. In this type of applications, it is critical to use neural networks that make predictions that are both fast and accurate. This means that, when designing these systems, we would like to tune different neural network parameters to jointly minimize two objectives: 1) the <strong>prediction error</strong> on some validation data and 2) the <strong>prediction speed</strong>. The parameters to tune, also called the <strong>design parameters</strong>, could be the number of hidden layers, the number of neurons per layer, learning rates, regularization parameters, etc.</p><p>Minimizing the prediction error and the prediction speed involves solving a <strong>multi-objective</strong> optimization problem. In this type of problems, in general, there is no single optimal solution that jointly minimizes the different objectives. Instead, there is a collection of optimal solutions called the <strong>Pareto front</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*EaGKNh1EUnofxF6K." /><figcaption>Example of Pareto front. Each point in the plot shows the objective values (f1 and f2) obtained for a particular setting of the design parameters. The points highlighted in blue are in the Pareto front.</figcaption></figure><p>The points in the Pareto front attain an optimal trade-off between objectives in the sense that you cannot move from one point in the Pareto front to another arbitrary point by improving one of the objectives without deteriorating the other. The figure above shows an example of a Pareto front for two objectives f1 and f2. Each point in the plot shows the objective values obtained for a particular choice of the design parameters. When solving a multi-objective optimization problem we are interested in finding values of the design parameters that map to points in the Pareto front. Once we have found the points in the Pareto front, we can make a decision and choose one that satisfies our requirements.</p><p>In the neural network tuning problem, the evaluation of the prediction error for a particular setting of the design parameters is a computationally expensive process. The reason for this is that, before being able to compute the validation error, we have to first train the neural network on large amounts of training data. With large neural networks and massive datasets this process can take many hours or even days. This means that, ideally, we would like to find the Pareto front using a small number of objective evaluations. This can be done by using<strong> Bayesian optimization</strong> (BO) techniques. These methods use the predictions of a model to efficiently search for the solution of expensive optimization problems. In this post, I will describe how to use the BO method Predictive Entropy Search for Multi-objective Optimization (<strong>PESMO</strong>)</p><p>Hernández-Lobato D., Hernández-Lobato J. M., Shah A. and R. P. Adams.<br><strong>Predictive Entropy Search for Multi-objective Bayesian Optimization</strong>,<br>In ICML, 2016. <a href="http://arxiv.org/pdf/1511.05467v3.pdf">[pdf]</a> [<a href="https://github.com/HIPS/Spearmint/tree/PESM">Spearmint code</a>]</p><p>for finding deep neural networks with optimal prediction error vs. prediction speed on the MNIST dataset. PESMO is the current state-of-the-art technique for multi-objective optimization with expensive objective functions. For more information about BO methods the reader can have a look at [1,2].</p><h3>Finding networks that produce fast and accurate predictions on MNIST</h3><p>PESMO is implemented in the PESM branch of the spearmint tool, which is publicly available at the link shown above. After installing the PESM branch of spearmint in our system, we have to specify the multi-objective optimization problem that we want to solve. In our case, we will be tuning the following design parameters in our neural networks:</p><ol><li>The number of hidden layers.</li><li>The number of neurons per hidden layer.</li><li>The level of dropout.</li><li>The learning rate to use with the ADAM optimizer [3].</li><li>The L1 weight penalty.</li><li>The L2 weight penalty.</li></ol><p>This information is provided to spearmint in a file called <strong>config.json</strong>, which contains details of the optimization problem such as the variables to be tuned, their maximum and minimum values, how many objectives we will be optimizing, how to evaluate the objectives, etc. We will be using the following config.json file:</p><pre>{<br>    &quot;language&quot;        : &quot;PYTHON&quot;,<br>    &quot;random_seed&quot;     : 1,<br>    &quot;grid_size&quot;       : 1000,<br>    &quot;max_finished_jobs&quot;       : 100,<br>    &quot;experiment-name&quot; : &quot;MNIST_neural_networks&quot;,<br>    &quot;moo_use_grid_only_to_solve_problem&quot; : true,<br>    &quot;moo_grid_size_to_solve_problem&quot; : 1000,<br>    &quot;pesm_use_grid_only_to_solve_problem&quot; : true,<br>    &quot;likelihood&quot;      : &quot;GAUSSIAN&quot;,<br>    &quot;acquisition&quot;     : &quot;PESM&quot;,<br>    &quot;pesm_pareto_set_size&quot;      : 50,<br>    &quot;pesm_not_constrain_predictions&quot;     : false,<br>    &quot;main_file&quot;       : &quot;evaluate_objectives&quot;,<br>    &quot;variables&quot;: {<br>        &quot;n_neurons&quot;: {<br>            &quot;type&quot;: &quot;FLOAT&quot;,<br>            &quot;size&quot;:  1,<br>            &quot;min&quot;:  50,<br>            &quot;max&quot;:  500<br>        }, &quot;n_hidden_layers&quot;: {<br>            &quot;type&quot;: &quot;FLOAT&quot;,<br>            &quot;size&quot;: 1,<br>            &quot;min&quot;:  1,<br>            &quot;max&quot;:  3<br>        }, &quot;prob_drop_out&quot;: {<br>            &quot;type&quot;: &quot;FLOAT&quot;,<br>            &quot;size&quot;: 1,<br>            &quot;min&quot;:  0,<br>            &quot;max&quot;:  0.9<br>        }, &quot;log_learning_rate&quot;: {<br>            &quot;type&quot;: &quot;FLOAT&quot;,<br>            &quot;size&quot;: 1,<br>            &quot;min&quot;: -20,<br>            &quot;max&quot;:  0<br>        }, &quot;log_l1_weight_reg&quot;: {<br>            &quot;type&quot;: &quot;FLOAT&quot;,<br>            &quot;size&quot;: 1,<br>            &quot;min&quot;: -20,<br>            &quot;max&quot;:  0<br>        }, &quot;log_l2_weight_reg&quot;: {<br>            &quot;type&quot;: &quot;FLOAT&quot;,<br>            &quot;size&quot;: 1,<br>            &quot;min&quot;: -20,<br>            &quot;max&quot;:  0<br>        }<br>    },<br>    &quot;tasks&quot;: {<br>      &quot;error_task&quot; : {<br>          &quot;type&quot;        : &quot;OBJECTIVE&quot;,<br>          &quot;fit_mean&quot;    : false,<br>          &quot;transformations&quot;     : [],<br>          &quot;group&quot;    : 0<br>      },<br>      &quot;time_task&quot; : {<br>          &quot;type&quot;        : &quot;OBJECTIVE&quot;,<br>          &quot;transformations&quot; : [<br>                {&quot;IgnoreDims&quot; : { &quot;to_ignore&quot; : [ 2, 3, 4, 5 ]   }}<br>                ],<br>          &quot;fit_mean&quot;    : false,<br>          &quot;group&quot;    : 0<br>      }<br>    }<br>}</pre><p>I will describe a few important entries in this file. The entry <strong>main_file</strong> contains the name of a python script that will be called to evaluate the different objectives. The entry <strong>max_finished_jobs</strong> takes value 100. This indicates spearmint to evaluate the different objectives 100 times and then finish. The entry <strong>variables</strong> contains a description of the design parameters, with their maximum and minimum values. The entry <strong>tasks</strong> contains a description of the objectives to be optimized. Note that the prediction time does not depend on the parameters dropout, learning rate, L1 penalty and L2 penalty. We inform spearmint of this by using the entry <strong>IgnoreDims</strong> in the transformations of the objective for the prediction time objective.</p><p>The next step is to write the python script that will evaluate the different objectives. We have already told spearmint that this file is called <strong>evaluate_objectives.py</strong>, which contains the following code based on Keras [4]:</p><pre>import numpy as np<br>import time<br>from keras.datasets import mnist<br>from keras.models import Sequential<br>from keras.layers.core import Dense, Dropout, Activation<br>from keras.optimizers import Adam<br>from keras.utils import np_utils<br>from keras.regularizers import l1l2, l2, activity_l2<br><br>def build_model(params):<br><br>    n_hidden_layers = int(np.round(params[&#39;n_hidden_layers&#39;][ 0 ]))<br>    n_neurons = int(np.round(params[&#39;n_neurons&#39;][ 0 ]))<br>    log_l1_weight_reg = params[&#39;log_l1_weight_reg&#39;][ 0 ]<br>    log_l2_weight_reg = params[&#39;log_l2_weight_reg&#39;][ 0 ]<br>    prob_drop_out = \<br>        float(params[&#39;prob_drop_out&#39;][ 0 ].astype(&#39;float32&#39;))<br>    log_l_rate = params[&#39;log_learning_rate&#39;][ 0 ]<br><br>    model = Sequential()<br>    model.add(Dense(n_neurons, input_shape = \<br>        (784,), W_regularizer=l1l2(l1 = np.exp(log_l1_weight_reg), \<br>        l2 = np.exp(log_l2_weight_reg))))<br>    model.add(Activation( &#39;relu&#39;))<br>    model.add(Dropout(prob_drop_out))<br>    for i in range(n_hidden_layers - 1):<br>        model.add(Dense(n_neurons, \<br>            W_regularizer=l1l2(l1 = np.exp(log_l1_weight_reg), \<br>            l2 = np.exp(log_l2_weight_reg))))<br>        model.add(Activation( &#39;relu&#39;))<br>        model.add(Dropout(prob_drop_out))<br>    n_classes = 10<br>    model.add(Dense(n_classes))<br>    model.add(Activation( &#39;softmax&#39;))<br>    adam = Adam(lr=np.exp(log_l_rate), beta_1=0.9, \<br>        beta_2=0.999, epsilon=1e-08)<br>    model.compile(loss=&#39;categorical_crossentropy&#39;, optimizer=adam)<br><br>    return model<br><br>def evaluate_error_model(X_train, Y_train, X_test, Y_test, \<br>    X_val, Y_val, params):<br><br>    nb_epoch = 150<br>    batch_size = 4000<br>    model = build_model(params)<br>    model.fit(X_train, Y_train, batch_size = \<br>        batch_size, nb_epoch=nb_epoch, \<br>        show_accuracy=True, verbose=2, \<br>        validation_data=(X_val, Y_val))<br>    loss, score = model.evaluate(X_val, Y_val, \<br>        show_accuracy=True, verbose=0)<br>    print(&#39;Val error:&#39;, 1.0 - score)<br>    return np.log((1 - score) / score)<br><br>def evaluate_time_model(X_train, Y_train, X_test, Y_test, \<br>    X_val, Y_val, params):<br><br>    nb_epoch = 1<br>    batch_size = 500<br>    model = build_model(params)<br>    start = time.time()<br>    for i in range(100):<br>        predictions = \<br>            model.predict_classes(X_val, \<br>            batch_size=X_val.shape[ 0 ], verbose=0)<br>    end = time.time()<br>    print(&#39;Avg. Prediction Time:&#39;, (end - start) / 100.0)<br>    return (end - start) / 100.0<br><br>def main(job_id, params):<br><br>    nb_classes = 10<br>    (X_train, y_train), (X_test, y_test) = mnist.load_data()<br>    X_train = X_train.reshape(60000, 784)<br>    X_test = X_test.reshape(10000, 784)<br>    X_train = X_train.astype(&#39;float32&#39;)<br>    X_test = X_test.astype(&#39;float32&#39;)<br>    X_train /= 255<br>    X_test /= 255<br><br>    state = np.random.get_state()<br>    np.random.seed(0)<br>    suffle = np.random.permutation(60000)<br>    i_train = suffle[ 0 : 50000 ]<br>    i_val = suffle[ 50000 : 60000 ]<br>    np.random.set_state(state)<br>    X_val = X_train[ i_val, : ]<br>    y_val = y_train[ i_val ]<br>    X_train = X_train[ i_train, : ]<br>    y_train = y_train[ i_train ]<br><br>    Y_train = np_utils.to_categorical(y_train, nb_classes)<br>    Y_test = np_utils.to_categorical(y_test, nb_classes)<br>    Y_val = np_utils.to_categorical(y_val, nb_classes)<br><br>    evaluation = dict()<br>    evaluation[&#39;error_task&#39;] = \<br>        evaluate_error_model(X_train, \<br>        Y_train, X_test, Y_test, X_val, Y_val, params)<br>    evaluation[&#39;time_task&#39;] = \<br>        evaluate_time_model(X_train, Y_train, X_test, \<br>        Y_test, X_val, Y_val, params)<br><br>    return evaluation</pre><p>Note that in the function <strong>build_model</strong> we <strong>round to integers </strong>the variables n_neurons and n_hidden_layers, which are assumed to be continuous by spearmint. Also, in this script, the function <strong>evaluate_error_model</strong> returns the logit of the error and not the error itself. The logit transformation expands the range of variation of the error for values close to zero and will allow the Gaussian process models used by spearmint to better describe the collected data. Once we have finished writing the config.json and the evaluate_objectives.py files, we will use spearmint to solve the optimization problem. Let us assume that the current working directory contains the files config.json and evaluate_objectives.py. Then we run spearmint by typing</p><pre>$ python /path_to_spearmint_folder/main.py .</pre><p>where <strong>path_to_spearmint_folder</strong> is the route to the folder where we have installed the PESM branch of spearmint. Spearmint will then evaluate the two objectives 100 times. After each of these evaluations, spearmint produces a recommendation with several points that are expected to be in the Pareto front. In general, the quality of these recommendations increases with the amount of data collected by spearmint. Therefore, in general, we will only be interested in the last of these recommendations. All the recommendations are stored by spearmint in a mongodb database. We can extract the last recommendation by using the python script <strong>extract_recommendations.py</strong>:</p><pre>import os<br>import sys<br>import numpy as np<br><br>from spearmint.utils.parsing          import parse_config_file<br>from spearmint.utils.database.mongodb import MongoDB<br>from spearmint.tasks.input_space      import InputSpace<br><br>def main():<br><br>    options         = parse_config_file(&#39;.&#39;, &#39;config.json&#39;)<br>    experiment_name = options[&quot;experiment-name&quot;]<br>    input_space     = InputSpace(options[&quot;variables&quot;])<br>    db              = \<br>        MongoDB(database_address=options[&#39;database&#39;][&#39;address&#39;])<br><br>    i = 0<br>    recommendation = db.load(experiment_name, \<br>        &#39;recommendations&#39;, {&#39;id&#39; : i + 1})<br>    while recommendation is not None:<br>        params_last = \<br>            input_space.vectorify(recommendation[ &#39;params&#39; ])<br>        recommendation = \<br>            db.load(experiment_name, \<br>            &#39;recommendations&#39;, {&#39;id&#39; : i + 1})<br>        i += 1<br><br>    np.savetxt(&#39;pareto_front.txt&#39;, params_last, fmt = &#39;%e&#39;)<br><br>if __name__ == &#39;__main__&#39;:<br>    main()</pre><p>We store this script in the same folder as the config.json file and execute it using</p><pre>$ python extract_recommendations.py</pre><p>The script extract_recommendations.py stores the values of the design parameters that are estimated to be in the Pareto front in the file <strong>pareto_front.txt</strong>. We can then evaluate the objectives at the recommended points using the script evaluate_objectives.py and choose the one that best fits our requirements. The following figure shows the average Pareto front obtained with PESMO and with other existing BO methods over 50 repetitions of the optimization process. Overall, PESMO is able to find neural networks with better trade-offs between prediction accuracy and prediction speed than the alternative techniques. By visualizing the Pareto front, as shown in the figure, we can also make better decisions regarding which points from the Pareto front we would like to choose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*mQDRdrvt8aHzmwkp." /><figcaption>Comparison of PESMO with other BO techniques in the task of finding neural networks that make accurate and fast predictions on the MNIST dataset. Each axis represents a different objective and the curves show the Pareto front produced by each method. On average, the Pareto front recommended by PESMO dominates all the others.</figcaption></figure><h3>Decoupled evaluations</h3><p>In the previous example we have always evaluated the two objectives one after the other and at the same input location. However, in practice, these two functions can be evaluated in a decoupled way, that is, independently and at different locations. Note that, to evaluate the prediction error on the validation data, we have to train the neural network first. However, training is not necessary for evaluating the prediction time, which can be done using a neural network with arbitrary weight values. This allows us to decouple the evaluation of these two objectives. In this decoupled setting, at any given time we could choose to evaluate the prediction error or the prediction time. In general, considering an optimization problem with decoupled objectives as such can produce significant gains, especially when one of the objectives is expensive to evaluate and not very informative about the solution to the optimization problem. Interestingly, PESMO also allows for solving multi-objective optimization problems with decoupled objectives. More details about optimization problems with decoupled function evaluations can be found in the reference for PESMO shown above or in [5]. An example application of decoupled Bayesian optimization in the constrained setting is given in [6]. In this work Predictive Entropy Search with Constraints (<strong>PESC</strong>) [7] is used to optimize the accuracy a statistical machine translation system while ensuring that decoding speed exceeds a minimum value.</p><h3>References</h3><ol><li>Brochu, Eric, Vlad M. Cora, and Nando De Freitas. <strong>A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning</strong>, <em>arXiv preprint arXiv:1012.2599</em> (2010).</li><li>Shahriari, Bobak, et al. <strong>Taking the human out of the loop: A review of Bayesian optimization,</strong> <em>Proceedings of the IEEE</em> 104.1 (2016): 148–175.</li><li>Kingma, Diederik, and Jimmy Ba. <strong>Adam: A method for stochastic optimization</strong>. <em>arXiv preprint arXiv:1412.6980</em> (2014).</li><li><a href="http://keras.io/">http://keras.io</a></li><li>Hernández Lobato J. M., Gelbart M. A., Adams R. P, Hoffman M. W. and Ghahramani Z. <strong>A General Framework for Constrained Bayesian Optimization using Information-based Search</strong>, Journal of Machine Learning Research, (in press), 2016. <a href="http://arxiv.org/pdf/1511.09422.pdf">[pdf]</a></li><li>Daniel Beck, Adrià de Gispert, Gonzalo Iglesias, Aurelien Waite and Bill Byrne. <strong>Speed-Constrained Tuning for Statistical Machine Translation Using Bayesian Optimization</strong>. <em>arXiv preprint arXiv:1604.05073</em> (2016).</li><li>Hernández-Lobato J. M., Gelbart A. M., Hoffman M. W., Adams R. and Ghahramani Z. <strong>Predictive Entropy Search for Bayesian Optimization with Unknown Constraints</strong>, In ICML, 2015. [<a href="https://jmhldotorg.files.wordpress.com/2015/05/pesc-icml2015.pdf">pdf]</a> <a href="https://jmhldotorg.files.wordpress.com/2015/05/pesc-icml2015-supplement.pdf">[supp. material]</a> <a href="https://github.com/HIPS/Spearmint/tree/PESC">[python code]</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2cdc3b21b50" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/neural-networks-with-optimal-accuracy-and-speed-in-their-predictions-d2cdc3b21b50">Neural networks with optimal accuracy and speed in their predictions</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>