<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roman Ring on Medium]]></title>
        <description><![CDATA[Stories by Roman Ring on Medium]]></description>
        <link>https://medium.com/@Inoryy?source=rss-30cb90653dc4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*E-p51sVgBxsBxSUGNkQdiQ.png</url>
            <title>Stories by Roman Ring on Medium</title>
            <link>https://medium.com/@Inoryy?source=rss-30cb90653dc4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:16:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Inoryy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Next Generation of Machine Learning Tools]]></title>
            <link>https://medium.com/@Inoryy/the-next-generation-of-machine-learning-tools-b0b0ef7f74a?source=rss-30cb90653dc4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0b0ef7f74a</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Roman Ring]]></dc:creator>
            <pubDate>Tue, 21 Jan 2020 22:00:00 GMT</pubDate>
            <atom:updated>2020-01-22T19:10:33.285Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Let’s travel back to a simpler time when all everyone talked about in machine learning were SVMs and boosted trees, while Andrew Ng introduced neural networks as a neat party hat trick you would probably never use in practice.</p><p>The year is 2012, and computer-vision based competition ImageNet is set to be once again won by the newest ensemble of kernel methods. That is, of course, until a couple of researchers unveiled AlexNet, having almost two times lower error rate than the competition, by using what we now commonly refer to as “deep learning.”</p><p>Many people point to AlexNet as one of the most important scientific breakthroughs of the decade, certainly one that helped change the landscape of ML research. However, it does not take much to realize that under the hood, it is “just” a combination of prior iterative improvements, many dating back to the early nineties. At its core, AlexNet is “just” a modified LeNet with more layers, better weight initialization, activation function, and data augmentation.</p><h3>Tools in ML Research</h3><p>So what made AlexNet stand out so much? I believe the answer lies in the tools researchers had at their disposal, enabling them to run artificial neural networks on GPU accelerators, a relatively novel idea at the time. In fact, Alex Krizhevsky’s former colleagues recall that many meetings before the competition consisted of Alex describing his progress with the CUDA quirks and features.</p><p>Now let us travel back to 2015 when ML research article submissions started blowing up across the board, including (re-)emergence of many now promising approaches such as generative adversarial learning, deep reinforcement learning, meta-learning, self-supervised learning, federated learning, neural architecture search, neural differential equations, neural graph networks, and many more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*W_RY6jxfzmT01Gy5.png" /><figcaption>Image via <a href="https://medium.com/@dcharrezt/neurips-2019-stats-c91346d31c8f">Charrez, D. (2019)</a>.</figcaption></figure><p>One could claim that this is just a natural outcome of the AI hype. However, I believe a significant factor was the emergence of the second generation of general-purpose ML frameworks such as TensorFlow and PyTorch, along with NVIDIA going all-in on AI. The frameworks that existed before, such as Caffe and Theano, were challenging to work with, and awkward to extend, which slowed down the research and development of novel ideas.</p><h3>A Need for Innovation</h3><p>TensorFlow and PyTorch were undoubtedly a net positive, and the teams worked hard to improve the libraries. Recently, they delivered TensorFlow 2.0 with a more straightforward interface along with eager mode, and PyTorch 1.0 with JIT compilation of the computation graph as well as support for XLA based accelerators such as TPUs. However, these frameworks are also beginning to reach their limits, forcing researchers into some paths while closing doors on others, just like their predecessors.</p><p>High-profile DRL projects such as AlphaStar and OpenAI Five not only utilized large-scale computational clusters but also pushed the limits of deep learning architecture components by combining deep transformers, nested recurrent networks, deep residual towers, among others.</p><p>In his <a href="https://www.thetimes.co.uk/article/demis-hassabis-interview-the-brains-behind-deepmind-on-the-future-of-artificial-intelligence-mzk0zhsp8">interview with The Times newspaper</a>, Demis Hassabis has stated that DeepMind will be focusing on applying AI directly for scientific breakthroughs. We can already see a glimpse of that shift in direction with some of their recent Nature articles on neuroscience and protein folding. Even a brief skim through the publications is enough to see that the projects required some unconventional approaches when it comes to engineering.</p><p>At NeurIPS 2019, probabilistic programming and bayesian inference were hot topics, especially uncertainty estimation and causal inference. Leading AI researchers presented their visions on what the future of ML might look like. Notably, Yoshua Bengio described transitioning to <a href="https://slideslive.com/38921750/from-system-1-deep-learning-to-system-2-deep-learning">system 2 deep learning</a> with out-of-distribution generalization, sparse graph networks, and causal reasoning.</p><p>To summarize, some of the requirements for next-gen ML tools are:</p><ul><li>fine-grained control flow use</li><li>non-standard optimization loops</li><li>higher-order differentiation as a first-class citizen</li><li>probabilistic programming as a first-class citizen</li><li>support for multiple heterogeneous accelerators in one model</li><li>seamless scalability from a single machine to gigantic clusters</li></ul><p>Ideally, the tools should also maintain a clean, straightforward, and extensible API, enabling scientists to research and develop their ideas rapidly.</p><h3>The Next Generation</h3><p>The good news is that many candidates already exist today, emerging in response to the needs in scientific computing. From experimental projects like Zygote.jl to even specialized languages, e.g. Halide and DiffTaichi. Interestingly, many projects draw inspiration from the fundamental works done by researchers in the auto-diff community, which evolved in parallel to ML.</p><p>Many of them were featured at the recent NeurIPS 2019 <a href="https://program-transformations.github.io/">workshop on program transformations</a>. The two I am most excited about are S4TF and JAX. They both tackle the task of making differentiable programming into an integral part of the toolchain, but in their own ways, almost orthogonal to each other.</p><h3>Swift for TensorFlow</h3><p>As the name suggests, S4TF tightly integrates the TensorFlow ML framework with the Swift programming language. A vote of confidence for the project is that it is led by Chris Lattner, who has authored LLVM, Clang, and Swift itself.</p><p>Swift is a compiled programming language, and one of its primary selling points is the powerful type system that is static and inferred. What the last part means in simpler terms is that Swift encompasses ease of use in languages like Python with code validations and transformations at compile-time, e.g., as in C++.</p><pre>let a: Int = 1<br>let b = 2<br>let c = &quot;3&quot;</pre><pre>print(a + b)         // 3<br>print(b + c)         // compilation (!) error<br>print(String(b) + c) // 23</pre><p>Swift features enable the S4TF team to meet quite a few requirements in the next-generation list by having analysis, verification, and optimization of the computation graph executed with efficient algorithms during compilation.</p><p>Crucially, the handling of automatic differentiation is off-loaded to the compiler.</p><pre>struct Linear: Differentiable {<br>  var w: Float<br>  var b: Float</pre><pre>func callAsFunction(_ x: Float) -&gt; Float {<br>    return w * x + b<br>  }<br>}</pre><pre>let f = Linear(w: 1, b: 2)<br>let 𝛁f = gradient(at: f) { f in f(3.0) }<br>print(𝛁f) // TangentVector(w: 3.0, b: 1.0)</pre><pre>let 𝛁f2 = gradient(at: f) { f in f([3.0]) } <br>// compilation (!) error: cannot convert... <br>// value of type &#39;[Float]&#39; to expected argument type &#39;Float&#39;</pre><p>Of course, TensorFlow itself is very well supported in this case.</p><pre>import TensorFlow</pre><pre>struct Model: Layer {<br>    var conv = Conv2D&lt;Float&gt;(filterShape: (5, 5, 6, 16), <br>      activation: relu)<br>    var pool = MaxPool2D&lt;Float&gt;(poolSize: (2, 2), strides: (2, 2))<br>    var flatten = Flatten&lt;Float&gt;()<br>    var dense = Dense&lt;Float&gt;(inputSize: 16 * 5 * 5, <br>      outputSize: 100, activation: relu)<br>    var logits = Dense&lt;Float&gt;(inputSize: 100, outputSize: 10,<br>      activation: identity)</pre><pre><a href="http://twitter.com/differentiable">@differentiable</a><br>    func callAsFunction(_ input: Tensor&lt;Float&gt;) -&gt; Tensor&lt;Float&gt; {<br>        return input.sequenced(<br>          through: conv, pool, flatten, dense, logits)<br>    }<br>}</pre><pre>var model = Model()<br>let optimizer = RMSProp(for: model, learningRate: 3e-4, <br>                        decay: 1e-6)</pre><pre>for batch in CIFAR10().trainDataset.batched(128) {<br>  let (loss, gradients) = valueWithGradient(at: model) { <br>    model in softmaxCrossEntropy(logits: model(batch.data), <br>                                 labels: batch.label)<br>  }<br>  print(loss)<br>  optimizer.update(&amp;model, along: gradients)<br>}</pre><p>On the other hand, if a critical feature is proving to be difficult to implement, having intimate knowledge of the whole pipeline is particularly valuable. For example, the <a href="https://mlir.llvm.org/">MLIR</a> compiler framework is a direct result of the S4TF efforts.</p><p>While differentiable programming is the core goal, S4TF is much more than that with a plan to support the infrastructure for various next-gen ML tools such as debuggers. For example, imagine an IDE warning a user that the custom model computation always results in a zero gradient without even executing it.</p><p>Python has an incredible community built around scientific computing and the S4TF team has explicitly taken the time to embrace it via interoperability.</p><pre>import Python // All that is necessary to enable the interop.</pre><pre>let np = Python.import(&quot;numpy&quot;) // Can import any Python module.<br>let plt = Python.import(&quot;matplotlib.pyplot&quot;)</pre><pre>let x = np.arange(0, 10, 0.01)<br>plt.plot(x, np.sin(x)) // Can use the modules as if inside Python.<br>plt.show() // Will show the sin plot, just as you would expect.</pre><p>This project is a significant undertaking and still has some ways to go before being ready for production. However, this is a great time to give it a try for both engineers and researchers and potentially contribute to its development. Work on S4TF has already produced interesting scientific advancements at the intersection of programming language and auto-diff theory.</p><p>One thing that especially stands out for me about S4TF is <a href="https://github.com/tensorflow/community/blob/master/sigs/swift/CHARTER.md">their approach to community outreach</a>. For example, the core developers hold weekly design sessions, which are open for anyone interested to join and even participate.</p><p>To learn more about Swift for TensorFlow, here are some useful resources:</p><ul><li><a href="https://course.fast.ai/part2.html#lesson-13-basics-of-swift-for-deep-learning">Fast.ai’s Lessons 13 and 14</a></li><li><a href="https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md">Design Doc: Why Swift For TensorFlow?</a></li><li><a href="https://colab.research.google.com/github/tensorflow/swift/blob/master/docs/site/tutorials/model_training_walkthrough.ipynb">Model Training Tutorial</a></li><li><a href="https://colab.research.google.com/github/tensorflow/swift/blob/master/notebooks/blank_swift.ipynb">Pre-Built Google Colab Notebook</a></li><li><a href="https://github.com/tensorflow/swift-models">Swift Models for Popular Architectures in DL and DRL</a></li></ul><h3>JAX</h3><p>JAX is a collection of function transformations such as just-in-time compilation and automatic differentiation, implemented as a thin wrapper over XLA with an API that is essentially a drop-in replacement for NumPy and SciPy. In fact, one way to get started with JAX is to think of it as an accelerator backed NumPy.</p><pre>import jax.numpy as np</pre><pre># Will be seamlessly executed on an accelerator such as GPU/TPU.<br>x, w, b = np.ones((3, 1000, 1000))<br>y = np.dot(w, x) + b</pre><p>Of course, in reality, JAX is much more than that. To many, it might seem that the project appeared out of thin air, but the truth is that it is an evolution of over five years of research spanning across three projects. Notably, JAX emerged from <a href="https://github.com/hips/autograd">Autograd</a> — a research endeavor into AD of native program code — generalizing on its core ideas to support arbitrary transformations.</p><pre>def f(x):<br>  return np.where(x &gt; 0, x, x / (1 + np.exp(-x)))</pre><pre># Note: same singular style for the API entry points.<br>jit_f = jax.jit(f) # Will be 10-100x faster.<br>grad_f = jax.grad(f) # Will work as expected.</pre><p>Aside from the grad and jit discussed above, there are two more excellent examples of JAX transformations, helping users to batch-process their data via auto-vectorization of batch dimension ( vmap) or multiple devices ( pmap).</p><pre>a = np.ones((100, 300))</pre><pre>def g(vec):<br>  return np.dot(a, vec)</pre><pre># Suppose `z` is a batch of 10 samples of 1 x 300 vectors.<br>z = np.ones((10, 300))</pre><pre>g(z) # Will not work due to (batch) dimension mismatch<br># (100x300 x 10x300).</pre><pre>vec_g = jax.vmap(g)<br>vec_g(z) # Will work, efficiently through batch dimension.</pre><pre># Manual solution requires &quot;playing&quot; with matrix transpositions.<br>np.dot(a, z.T)</pre><p>These features might seem confusing at first, but after some practice, they turn into an irreplaceable part of a researcher’s toolbox. They have even inspired recent development of similar functionality in both <a href="https://www.tensorflow.org/api_docs/python/tf/vectorized_map">TensorFlow</a> and <a href="https://twitter.com/apaszke/status/1219886260296261632">PyTorch</a>.</p><p>For the time being, JAX authors seem to be sticking to their core competency when it comes to developing new features. Of course, a reasonable approach but is also the cause for one of its main drawbacks: lack of built-in neural network components, aside from the proof-of-concept <a href="https://github.com/google/jax/blob/master/jax/experimental/stax.py">Stax</a>.</p><p>Adding higher-level features is something where end-users can potentially step in and contribute, and given JAX’s solid foundation, the task might be easier than it seems. For example, there are now two “competing” libraries built on top of JAX, both developed by Google researchers, with differing approaches: <a href="https://github.com/google/trax">Trax</a> and <a href="https://github.com/google-research/flax/tree/prerelease">Flax</a>.</p><pre># Trax approach is functional.<br># Note: params are stored outside and `forward` is &quot;pure&quot;.</pre><pre>import jax.numpy as np<br>from trax.layers import base</pre><pre>class Linear(base.Layer):<br>  def __init__(self, num_units, init_fn):<br>    super().__init__()<br>    self.num_units = num_units<br>    self.init_fn = init_fn</pre><pre>def forward(self, x, w):<br>    return np.dot(x, w)</pre><pre>def new_weights(self, input_signature):<br>    w = self.init_fn((input_signature.shape, self._num_units))<br>    return w</pre><pre># Flax approach is object-oriented, closer to PyTorch style.</pre><pre>import jax.numpy as np<br>from flax import nn</pre><pre>class Linear(nn.Module):<br>  def apply(self, x, num_units, init_fn):<br>    W = self.param(&#39;W&#39;, (x.shape[-1], num_units), init_fn)<br>    return np.dot(x, W)</pre><p>Even though some might prefer a singular way, endorsed by core developers, having a diversity of methods is a good indicator that the technology is sound.</p><p>There are also some directions in research where JAX features especially shine. For example, in meta-learning, one common approach to training the meta-learner is by computing the gradients of the inputs. An alternative method for computing gradients — forward-mode auto-differentiation — is necessary to solve this task efficiently, which is supported out-of-the-box in JAX but is either non-existent or an experimental feature in other libraries.</p><p>JAX is perhaps more polished and production-ready than its S4TF counter-part and some of the recent developments coming out of Google Research rely on it, such as Reformer — a memory-efficient Transformer model capable of handling context windows of a million words while fitting on a consumer GPU, and Neural Tangents — a library for complex neural networks of infinite width.</p><p>The library is further embraced by the broader scientific computing community, used for works in molecular dynamics and probabilistic programming, among others.</p><p>To get started with JAX and for further reading, please review the following:</p><ul><li><a href="https://slideslive.com/38922046/program-transformations-for-ml-3">Talk: Overview by Skye Wanderman-Milne</a> (starts at 44:26)</li><li><a href="https://jax.readthedocs.io/en/latest/notebooks/quickstart.html">Notebook: Quickstart, going over fundamental features</a></li><li><a href="https://github.com/google/jax/tree/master/cloud_tpu_colabs">Notebook: Cloud TPU Playground</a></li><li><a href="https://colinraffel.com/blog/you-don-t-know-jax.html">Blog: You don’t know JAX</a></li><li><a href="https://rlouf.github.io/post/jax-random-walk-metropolis/">Blog: Massively parallel MCMC with JAX</a></li><li><a href="https://blog.evjang.com/2019/11/jaxpt.html">Blog: Differentiable Path Tracing on the GPU/TPU</a></li></ul><h3>Conclusion</h3><p>ML research is starting to hit the limits of the tools we currently have at our disposal, but some new and exciting candidates are right around the corner, such as JAX and S4TF. If you feel yourself to be more of an engineer than a researcher and wonder whether there is even a place for you at the ML table, hopefully, the answer is clear: right now is the perfect time to get into it. Moreover, you have an opportunity to participate on the ground floor of the next generation of ML tools!</p><p>Note that this does not mean TensorFlow or PyTorch are going anywhere, not in the near future. There is still much value in these mature, battle-tested libraries. After all, both JAX and S4TF have parts of TensorFlow under their hoods. But if you are about to start a new research project or if you feel that you are working around library limitations more than on your ideas, then maybe give them a try!</p><p><em>Originally published at </em><a href="http://inoryy.com/post/next-gen-ml-tools/"><em>http://inoryy.com</em></a><em> on January 21, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0b0ef7f74a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepMind, StarCraft II, and The Next Big Thing in AI | Roman Ring]]></title>
            <link>https://medium.com/@Inoryy/deepmind-starcraft-ii-and-the-next-big-thing-in-ai-roman-ring-8ad114c2a2cc?source=rss-30cb90653dc4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ad114c2a2cc</guid>
            <category><![CDATA[deepmind]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[starcraft-ii]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Roman Ring]]></dc:creator>
            <pubDate>Wed, 23 Jan 2019 22:00:00 GMT</pubDate>
            <atom:updated>2019-01-26T23:11:12.561Z</atom:updated>
            <content:encoded><![CDATA[<h3>DeepMind, StarCraft II, and The Next Big Thing in AI</h3><p>Couple of days ago out of the blue DeepMind announced a StarCraft II related event, with many of the employees being quite excited about it on twitter. In just a few hours we will see what DeepMind has in store for us, but in the meantime let’s take a step back and review how we got here and why it is so important.</p><p><strong>Update</strong>: the (amazing) event has finished and DeepMind have released a <a href="https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/">very detailed write-up</a>. See below for my thoughts on the event and the write-up.</p><h3>DeepMind</h3><p>At the end of 2013, DeepMind was a small and relatively unknown startup, founded three years prior with one simple goal: “to solve intelligence”. Although not yet famous with the general public, they have made a lot of noise in the world of deep reinforcement learning by introducing <a href="https://deepmind.com/research/dqn/">DQN</a> — an algorithm, capable of matching or even surpassing humans at playing the Atari video games, while observing the game from raw pixels similar to how a human would. In the early 2014 DeepMind was bought by Google Inc. for $500 million, many speculating that deal was ensured by this one achievement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/0*loIEHaZoS95LCYv1.gif" /></figure><p>In March of 2016 DeepMind did it again, with their <a href="https://deepmind.com/research/alphago/">AlphaGo</a> program decisively winning against Go world champion Lee Sedol, who held the title for a great number of years. While in hindsight this seems as a sure thing, three years ago this win came almost out of nowhere, with many researchers at the time were predicting that AI will not match human experts for another 10 years at least.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sAvBAK2qFBY1E3Z8.jpg" /></figure><h3>StarCraft II</h3><p>Shortly after DeepMind’s AlphaGo victory, DeepMind were already prepared to announce the next challenge they will be tackling: the StarCraft II video game. StarCraft II is a real-time strategy game that requires making split second decisions on strategical, tactical, and economical levels, with short-term and long-term goals, throughout the whole duration of the game. Together with its predecessor StarCraft: Brood War, it has a rich competitive e-sports history spanning over 20 years, and is actively played by millions to this day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cH9_9sW85Jfjh2R7.png" /></figure><p>With much bigger state and action spaces, real-time component, partial observability, and longer game duration, this is a much more difficult task than any attempted before. In fact many AI researchers have actually attempted to tackle it, notably the StarCraft: Brood War bot <a href="https://arstechnica.com/gaming/2011/01/skynet-meets-the-swarm-how-the-berkeley-overmind-won-the-2010-starcraft-ai-competition/">OverMind</a> was a collaborative effort of many talented scientists. Yet, when pitted against human players, the AI champion would be easily defeated even by relative novices to the game.</p><p>This massive challenge alone wasn’t enough for DeepMind researchers, they decided to go one step further. They wanted to not only challenge human experts, but do so “on their terms”. The AI will be observing the game in a similar way to humans, through image-like features, and will be acting by emulating keyboard and mouse commands as close as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vwzQ9de926FElzvK.jpg" /></figure><p>Furthermore, the AI is limited in many aspects to match human experiences, e.g. it is allowed to take a limited number of actions per minute (APM) and can only access the same information a human would see on the screen or minimap. This means that the AI must learn to use in-game camera in order to effectively play.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/0*YINFM8Azhf5FJAd2.gif" /></figure><h3>The Next Big Thing in AI</h3><p>A StarCraft II player might wonder what is special about this if there were relatively good AIs built into the game from release. The AI you encounter in-game acts based on some pre-defined set of rules, which means it can only play as well as the person who programmed it, reacting only to foreseen events. This makes the AI inflexible, and easy to exploit, among other things.</p><p>In contrast, the AI DeepMind is working on learns the game by itself, essentially from scratch. At each step all it has is the game state it observed, the actions it can take, and some reward stimulus it received for a previous action. In general the approach is called reinforcement learning and <a href="http://inoryy.com/post/tensorflow2-deep-reinforcement-learning/">here</a> I give a brief overview of the field.</p><p>To get a feel for what it’s like for an RL agent to “learn” the game, here is a video of an agent attempting to tackle a set of minigames released by DeepMind. On the left you can see an agent that just started the process and is exploring all the various actions it can take, and on the right is the same agent after a fair bit of such experiments (about 50 million steps on average).</p><p>An outside reader might wonder why AI researchers focus so much on games to begin with. The answer is simple: real-world is just too complex for an AI system at this point — one must first learn to walk before he gets to fly. Specifically, games have easily defined rules that can be described to an AI system and are typically easy to run in parallel on a computer, speeding up the learning process.</p><p>This however does not mean that current work will be useless in the future. For example, while “playing” with StarCraft II, DeepMind researchers produced a number of general-purpose results such as the <a href="https://deepmind.com/blog/population-based-training-neural-networks/">population based training</a>, and <a href="https://openreview.net/forum?id=HkxaFoC9KQ">relational deep reinforcement learning</a>. So while developing an AI capable of winning in StarCraft II with human-like restrictions <em>probably</em> won’t give us <a href="https://en.wikipedia.org/wiki/Artificial_general_intelligence">AGI</a>, it will definitely bring us an inch closer.</p><h3>Predictions and Speculations</h3><p>In this section I will attempt to predict some of the things we will see at the event, along with the nitty-gritty details “under the hood” of the AI, which might get a bit technical. Note that I have no insider information, this is simply my guesses.</p><p>For the event itself I believe it will be a Protoss vs Protoss showmatch between DeepMind’s AI and some high level player, either ex-pro or possibly even current pro, however not best of the best tier. I would be <em>really</em> surprised if they are already at a level to challenge world champions.</p><p>The core algorithm they will employ will be an actor-critic variant called <a href="https://deepmind.com/blog/impala-scalable-distributed-deeprl-dmlab-30/">IMPALA</a>, fused with attention mechanism as described in <a href="https://openreview.net/forum?id=HkxaFoC9KQ">Relational DRL</a>. Network architecture will have residual + convolutional state encoder component and a number of Conv LSTM blocks generating the policy. It will also have a fair bit of imitation learning based pre-training prior to the DRL loop.</p><p>The IMPALA algorithm is essentially a distributed, asyncronous version of A2C I’ve described in <a href="http://inoryy.com/post/tensorflow2-deep-reinforcement-learning/">this</a> blog with a number of fixes (e.g. importance weighting) to address its “off-policyness” - the fact that samples are gathered with a different policy than the one being currently trained. If you’re familiar with <a href="https://blog.openai.com/openai-five/">openAI Five</a> then the core idea is similar to their asyncronous PPO: a (massively) distributed advantage actor-critic variant with tricks to correct for the distributional drift.</p><p>The attention mechanism has revolutionized the world of NLP, especially after the “<a href="https://arxiv.org/abs/1706.03762">Attention is All You Need</a>” article came out. In short, if we’re dealing with sequential input / output then attention mechanism acts as importance weight for items in the given sequence when generating the next output. In NLP this is most often used for machine translation to give attention to words at different parts of the sentence. In StarCraft II this could be used to ensure the agent can quickly switch between tactical and strategical decision making. DeepMind has recently applied it to DRL in their <a href="https://openreview.net/forum?id=HkxaFoC9KQ">Relational DRL</a> article.</p><p>The residual + convolutional architecture makes sense to use given our state space, whereas Conv LSTM cells (NB! different from <em>CNN</em> LSTM) are starting to gain traction in DRL world, especially when observations are rich with spatial information.</p><p>During Blizzcon 2018 presentation Oriol Vinyals mentioned that they’ve used imitation learning for their agent - and it of course make sense, given that Blizzard is providing free access (to everybody!) to their massive dataset of replays. Of course it would be impossible to succesfully train a full agent from replays alone, but I think it’s reasonable to use it for NN weights pre-training.</p><p>I also wouldn’t be surprised if instead of fully end-to-end approach DeepMind will rely on something modular, e.g. as described <a href="https://arxiv.org/abs/1811.03555">here</a>, where individual modules are responsible for specific subsets of the game. Specifically, perhaps DeepMind relies on pre-defined rulesets for scouting and makes use of a simplified game engine for battle simulations that are used in MCTS solver to determine whether they should attack or fall back.</p><h3>Conclusion</h3><p>Hopefully you’re now as excited as I am about the upcoming event. Whether the AI is fully end to end or not and whether they will be able to win vs human experts or not, it is still a massive endeavor and should provide for a good show. And if you’d like to get into developing StarCraft II based AIs yourself, join our <a href="https://discordapp.com/invite/Emm5Ztz">SC2AI community on discord</a>!</p><h3>Post-Event Write-Up</h3><p>The stream was very exciting to watch, both as a player and a researcher. It was funny to see AlphaStar opt for wild strategies and then come out on top. It was also interesting to see it make mistakes such as killing its own units, very AI and human-like behavior at the same time.</p><p>To me the most impressive was the micro, and not just due to its ability to make split second decisions. The way AlphaStar knew how to pull back damaged stalkers to regenerate shields, the way it pulled its workers when it saw Oracles — really mind boggling that a single end-to-end neural network is capable of such a rich variety of tactical and somewhat long-term decision-making.</p><p>However, AlphaStar is still quite far from conquering StarCraft II universe. First, while Mana is no doubt a great player, he is not quite world champion caliber. Second, this is still a single matchup, whereas any human player would be expected to play vs all three races on the same level. Third, I’m not sure how I feel about having players go against a pool of AlphaStar(s) — I think it definitely makes sense to use for training, but during inference I’d prefer to see a single version used throughout the matches. Overall I would say AlphaStar right now is closer to the AlphaGo version that played vs Fan Hui than the one that won vs Lee Sedol.</p><p>Seems that I’ve correctly predicted the match-up and level of play, along with some of the approaches. Specifically, AlphaStar does indeed rely on imitation learning, IMPALA, and attention mechanism, though not quite as described in Relational DRL article. They also indeed use LSTM, but I am not so sure with regards to convolutional layers - there seems to be a bit of confusion as to what interface they ended up using. I’ve also briefly mentioned population based training - seems that DeepMind uses an advanced variant of it, hopefully we will see an article about it soon.</p><p>Of the things I’ve missed is the <a href="https://arxiv.org/abs/1706.03762">transformer</a> body, which is a state-of-the-art architecture in machine translation. Very surprised to see it applied in DRL. They also use a relatively novel baseline for the advantage function in the PG loss, which they pulled from the <a href="https://www.cs.ox.ac.uk/people/shimon.whiteson/pubs/foersteraaai18.pdf">Counterfactual Multi-Agent Policy Gradients</a> article. This is noteworthy because for the longest time the value estimate of next state for baseline was the go-to approach of pretty much everybody in DRL.</p><p>Finally, they apply <a href="https://arxiv.org/abs/1506.03134">pointer network</a> to the policy output, most likely as an efficient way to deal with variable length of action arguments. To me the use of pointer networks was quite surprising and somewhat ironic - I have actually <a href="http://inoryy.com/files/pointer_networks_essay.pdf">written an essay</a> on this article and while it was an interesting subject, it never crossed my mind it could be applied in such a way to DRL policies. Although in retrospect I guess it makes sense.</p><p>During training they also rely on <a href="http://proceedings.mlr.press/v80/oh18b/oh18b.pdf">self-imitation</a> and experience replay, which is quite interesting - seems they have finally perfected the combination of actor-critic methods, which are traditionally seen as on-policy, with the benefits of off-policy algorithms. Finally, they use <a href="https://arxiv.org/pdf/1511.06295.pdf">policy distillation</a> which is probably how they were able to fit the final agents into a single machine for inference.</p><p>If by now your head is spinning from all the terminology, don’t worry — mine is too. The takeaway message is that it took an impressive amount of very advanced approaches to achieve the level of play we have seen today and I am curious to see what happens next.</p><p><em>Originally published at </em><a href="http://inoryy.com/post/deepmind-starcraft2-next-big-thing-ai/"><em>inoryy.com</em></a><em> on January 24, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ad114c2a2cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Reinforcement Learning with TensorFlow 2.0]]></title>
            <link>https://medium.com/@Inoryy/deep-reinforcement-learning-with-tensorflow-2-0-d8e62102680d?source=rss-30cb90653dc4------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8e62102680d</guid>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Roman Ring]]></dc:creator>
            <pubDate>Sat, 19 Jan 2019 22:00:00 GMT</pubDate>
            <atom:updated>2020-01-15T20:40:49.245Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deep Reinforcement Learning With TensorFlow 2.1</h3><p>In this tutorial, I will give an overview of the TensorFlow 2.x features through the lens of deep reinforcement learning (DRL) by implementing an advantage actor-critic (A2C) agent, solving the classic CartPole-v0 environment. While the goal is to showcase TensorFlow 2.x, I will do my best to make DRL approachable as well, including a birds-eye overview of the field.</p><p>In fact, since the main focus of the 2.x release is making life easier for the developers, it’s a great time to get into DRL with TensorFlow. For example, the source code for this blog post is under 150 lines, including comments!<br>Code is available on GitHub <a href="https://github.com/inoryy/tensorflow2-deep-reinforcement-learning">here</a> and as a notebook on Google Colab <a href="https://colab.research.google.com/drive/1XoHmGiwo2eUN-gzSVLRvE10fIf_ycO1j">here</a>.</p><h3>Setup</h3><p>To follow along, I recommend setting up a separate (virtual) environment.<br>I prefer <a href="https://www.anaconda.com/download">Anaconda</a>, so I’ll illustrate with it:</p><pre>&gt; conda create -n tf2 python=3.7<br>&gt; conda activate tf2<br>&gt; pip install tensorflow=2.1</pre><p>Now verify everything works as expected:</p><pre>&gt;&gt;&gt; import tensorflow as tf<br>&gt;&gt;&gt; print(tf.__version__)<br>2.1.0</pre><p>Note that we are now in eager mode by default!</p><pre>&gt;&gt;&gt; print(tf.executing_eagerly())<br>True<br>&gt;&gt;&gt; print(&quot;1 + 2 + 3 + 4 + 5 =&quot;, tf.reduce_sum([1, 2, 3, 4, 5]))<br>1 + 2 + 3 + 4 + 5 = tf.Tensor(15, shape=(), dtype=int32)</pre><p>If you are not yet familiar with eager mode, then, in essence, it means that computation executes at runtime, rather than through a pre-compiled graph. You can find a good overview in the <a href="https://www.tensorflow.org/tutorials/eager/eager_basics">TensorFlow documentation</a>.</p><h4>GPU Support</h4><p>One great thing about specifically TensorFlow 2.1 is that there is no more hassle with separate CPU/GPU wheels! TensorFlow now supports both by default and targets appropriate devices at runtime.</p><p>The benefits of Anaconda are immediately apparent if you want to use a GPU. Setup for all the necessary CUDA dependencies is just one line:</p><pre><strong>&gt;</strong> conda install cudatoolkit=10.1</pre><p>You can even install different CUDA toolkit versions in separate environments!</p><h3>Reinforcement Learning</h3><p>Generally speaking, reinforcement learning is a high-level framework for solving sequential decision-making problems. An RL agent navigates an environment by taking actions based on some observations, receiving rewards as a result. Most RL algorithms work by maximizing the expected total rewards an agent collects in a trajectory, e.g., during one in-game round.</p><p>The output of an RL algorithm is a policy – a function from states to actions. A valid policy can be as simple as a hard-coded no-op action, but typically it represents a conditional probability distribution of actions given some state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sbkb2Wh2d-azWe-q.png" /><figcaption><em>Figure: A general diagram of the RL training loop.<br>Image via </em><a href="http://web.stanford.edu/class/cs234/index.html"><em>Stanford CS234 (2019)</em></a><em>.</em></figcaption></figure><p>RL algorithms are often grouped based on their optimization loss function.</p><p>Temporal-Difference methods, such as Q-Learning, reduce the error between predicted and actual state(-action) values.</p><p>Policy Gradients directly optimize the policy by adjusting its parameters. Calculating gradients themselves is usually infeasible; instead, they are often estimated via monte-carlo methods.</p><p>The most popular approach is a hybrid of the two: actor-critic methods, where policy gradients optimize agent’s policy, and the temporal-difference method is used as a bootstrap for the expected value estimates.</p><h4>Deep Reinforcement Learning</h4><p>While much of the fundamental RL theory was developed on the tabular cases, modern RL is almost exclusively done with function approximators, such as artificial neural networks. Specifically, an RL algorithm is considered deep if the policy and value functions are approximated with neural networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*eM5Ar8XfQ5pPn8a_.jpg" /><figcaption><em>Figure: DRL implies ANN is used in the agent’s model.<br>Image via </em><a href="https://arxiv.org/abs/1810.04107"><em>Mohammadi et al (2018)</em></a><em>.</em></figcaption></figure><h4>(Asynchronous) Advantage Actor-Critic</h4><p>Over the years, several improvements were added to address sample efficiency and stability of the learning process.</p><p>First, gradients are weighted with returns: a discounted sum of future rewards, which resolves theoretical issues with infinite timesteps, and mitigates the credit assignment problem – allocate rewards to the correct actions.</p><p>Second, an advantage function is used instead of raw returns. Advantage is formed as the difference between the returns and some baseline, which is often the value estimate, and can be thought of as a measure of how good a given action is compared to some average.</p><p>Third, an additional entropy maximization term is used in the objective function to ensure the agent sufficiently explores various policies. In essence, entropy measures how <em>random</em> a given probability distribution is. For example, entropy is highest in the uniform distribution.</p><p>Finally, multiple workers are used in parallel to speed up sample gathering while helping decorrelate them during training, diversifying the experiences an agent trains on in a given batch.</p><p>Incorporating all of these changes with deep neural networks, we arrive at the two of the most popular modern algorithms: (asynchronous) advantage actor critic, or A3C/A2C for short. The difference between the two is more technical than theoretical. As the name suggests, it boils down to how the parallel workers estimate their gradients and propagate them to the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i0z9iihFZPAYrMNC.png" /><figcaption><em>Image via </em><a href="http://bit.ly/2uAJm2S"><em>Juliani A. (2016)</em></a><em>.</em></figcaption></figure><p>With this, we wrap up our tour of the DRL methods and move on to the focus of the blog post is more on the TensorFlow 2.x features. Don’t worry if you are still unsure about the subject; should become clearer with code examples.<br>If you want to learn more, one excellent resource is <a href="https://spinningup.openai.com/en/latest">Spinning Up in Deep RL</a>.</p><h3>Advantage Actor-Critic with TensorFlow 2.1</h3><p>Now that we are more or less on the same page, let’s see what it takes to implement the basis of many modern DRL algorithms: an actor-critic agent, described in the previous section. Without parallel workers (for simplicity), though most of the code would be the same.</p><p>As a testbed, we are going to use the <a href="https://gym.openai.com/envs/CartPole-v0/">CartPole-v0</a> environment. Somewhat simplistic, it is still a great option to get started. In fact, I often rely on it as a sanity check when implementing RL algorithms.</p><h4>Policy &amp; Value Models via Keras API</h4><p>First, we create the policy and value estimate NNs under a single model class:</p><pre>import numpy as np<br>import tensorflow as tf<br>import tensorflow.keras.layers as kl</pre><pre>class <strong>ProbabilityDistribution</strong>(tf.keras.Model):<br>  def <strong>call</strong>(self, logits, **kwargs):<br>    # Sample a random categorical action from the given logits.<br>    return tf.squeeze(tf.random.categorical(logits, 1), axis=-1)</pre><pre>class <strong>Model</strong>(tf.keras.Model):<br>  def <strong>__init__</strong>(self, num_actions):<br>    super().__init__(&#39;mlp_policy&#39;)<br>    # Note: no tf.get_variable(), just simple Keras API!<br>    self.hidden1 = kl.Dense(128, activation=&#39;relu&#39;)<br>    self.hidden2 = kl.Dense(128, activation=&#39;relu&#39;)<br>    self.value = kl.Dense(1, name=&#39;value&#39;)<br>    # Logits are unnormalized log probabilities.<br>    self.logits = kl.Dense(num_actions, name=&#39;policy_logits&#39;)<br>    self.dist = ProbabilityDistribution()</pre><pre>  def <strong>call</strong>(self, inputs, **kwargs):<br>    # Inputs is a numpy array, convert to a tensor.<br>    x = tf.convert_to_tensor(inputs)<br>    # Separate hidden layers from the same input tensor.<br>    hidden_logs = self.hidden1(x)<br>    hidden_vals = self.hidden2(x)<br>    return self.logits(hidden_logs), self.value(hidden_vals)</pre><pre>  def <strong>action_value</strong>(self, obs):<br>    # Executes `call()` under the hood.<br>    logits, value = self.predict_on_batch(obs)<br>    action = self.dist.predict_on_batch(logits)<br>    # Another way to sample actions:<br>    #   action = tf.random.categorical(logits, 1)<br>    # Will become clearer later why we don&#39;t use it.<br>    return np.squeeze(action, axis=-1), np.squeeze(value, axis=-1)</pre><p>And verify the model works as expected:</p><pre>import gym</pre><pre>env = gym.make(&#39;CartPole-v0&#39;)<br>model = Model(num_actions=env.action_space.n)</pre><pre>obs = env.reset()<br># No feed_dict or tf.Session() needed at all!<br>action, value = model.action_value(obs[None, :])<br>print(action, value) # [1] [-0.00145713]</pre><p>Things to note here:</p><ul><li>Model layers and execution path are defined separately</li><li>There is no “input” layer; model accepts raw numpy arrays</li><li>Two computation paths can exist in one model via functional API</li><li>A model can contain helper methods such as action sampling</li><li>In eager mode, everything works from raw numpy arrays</li></ul><h4>Agent Interface</h4><p>Now we can move on to the fun stuff — the agent class. First, we add a test method that runs through a full episode, keeping track of the rewards.</p><pre>class <strong>A2CAgent</strong>:<br>  def <strong>__init__</strong>(self, model):<br>    self.model = model</pre><pre>  def <strong>test</strong>(self, env, render=True):<br>    obs, done, ep_reward = env.reset(), False, 0<br>    while not done:<br>      action, _ = self.model.action_value(obs[None, :])<br>      obs, reward, done, _ = env.step(action)<br>      ep_reward += reward<br>      if render:<br>        env.render()<br>    return ep_reward</pre><p>Now we can check how much the agent scores with random weights:</p><pre>agent = A2CAgent(model)<br>rewards_sum = agent.test(env)<br>print(&quot;%d out of 200&quot; % rewards_sum) <strong># 18 out of 200</strong></pre><p>Not even close to optimal, time to get to the training part!</p><h4>Loss / Objective Function</h4><p>As I have described in the RL section, an agent improves its policy through gradient descent based on some loss (objective) function. In the A2C algorithm, we train on three objectives: improve policy with advantage weighted gradients, maximize the entropy, minimize value estimate errors.</p><pre>import tensorflow.keras.losses as kls<br>import tensorflow.keras.optimizers as ko</pre><pre>class <strong>A2CAgent</strong>:<br>  def <strong>__init__</strong>(self, model, lr=7e-3, value_c=0.5, entropy_c=1e-4):<br>    <em># Coefficients are used for the loss terms.</em><br>    self.value_c = value_c<br>    self.entropy_c = entropy_c</pre><pre>    self.model = model<br>    self.model.compile(<br>      optimizer=ko.RMSprop(lr=lr),<br>      <em># Define separate losses for policy and value estimate.</em><br>      loss=[self._logits_loss, self._value_loss])</pre><pre>  def <strong>test</strong>(self, env, render=False):<br>    <em># Unchanged from the previous section.</em><br>    ...</pre><pre>  def <strong>_value_loss</strong>(self, returns, value):<br>    <em># Typically MSE between value estimates and returns.</em><br>    return self.value_c * kls.mean_squared_error(returns, value)</pre><pre>  def<strong> _logits_loss(</strong>self, actions_and_advs, logits<strong>):</strong><br>    <em># A trick to input actions and advantages through the same API.</em><br>    actions, advantages = tf.split(actions_and_advs, 2, axis=-1)</pre><pre>    <em># Sparse categorical CE loss obj that supports sample_weight.<br>    # `from_logits` arg ensures normalized probabilities are used.</em><br>    weighted_sparse_ce =<br>      kls.SparseCategoricalCrossentropy(from_logits=True)</pre><pre>   <em> # Policy loss -- policy gradients, weighted by advantages.<br>    # Note: we calculate the loss on the actions actually taken.</em><br>    actions = tf.cast(actions, tf.int32)<br>    policy_loss = weighted_sparse_ce(actions, logits,<br>      sample_weight=advantages)</pre><pre>    <em># Entropy loss can be calculated as cross-entropy over itself.</em><br>    probs = tf.nn.softmax(logits)<br>    entropy_loss = kls.categorical_crossentropy(probs, probs)</pre><pre>    <em># We want to minimize policy and maximize entropy losses.<br>    # Here signs are flipped because the optimizer minimizes.</em><br>    return policy_loss - self.entropy_c * entropy_loss</pre><p>And we are done with the objective functions! Note how compact the code is: there are almost more comment lines than code itself.</p><h4>The Training Loop</h4><p>Finally, there is the train loop itself. It is relatively long, but fairly straightforward: collect samples, calculate returns and advantages, and train the model on them.</p><pre>class <strong>A2CAgent</strong>:<br>  def <strong>__init__</strong>(<br>    self, model, lr=7e-3, gamma=0.99, value_c=0.5, entropy_c=1e-4<br>  ):<br>    # `gamma` is the discount factor<br>    self.gamma = gamma<br>    # Unchanged from the previous section.<br>    ...<br>  <br>  def <strong>train</strong>(self, env, batch_sz=64, updates=250):<br>    # Storage helpers for a single batch of data.<br>    actions = np.empty((batch_sz,), dtype=np.int32)<br>    rewards, dones, values = np.empty((3, batch_sz))<br>    observations =<br>      np.empty((batch_sz,) + env.observation_space.shape)</pre><pre>    # Training loop: <br>    #   collect samples, send to optimizer, repeat updates times.<br>    ep_rewards = [0.0]<br>    next_obs = env.reset()<br>    for update in range(updates):<br>      for step in range(batch_sz):<br>        observations[step] = next_obs.copy()<br>        actions[step], values[step] =<br>          self.model.action_value(next_obs[None, :])<br>        next_obs, rewards[step], dones[step], _ = <br>          env.step(actions[step])</pre><pre>        ep_rewards[-1] += rewards[step]<br>        if dones[step]:<br>          ep_rewards.append(0.0)<br>          next_obs = env.reset()<br>          logging.info(&quot;Episode: %03d, Reward: %03d&quot; % (<br>            len(ep_rewards) - 1, ep_rewards[-2]))</pre><pre>      _, next_value = self.model.action_value(next_obs[None, :])</pre><pre>      returns, advs = self._returns_advantages(<br>        rewards, dones, values, next_value)<br>      # A trick to input actions and advantages through same API.<br>      acts_and_advs = <br>        np.concatenate([actions[:, None], advs[:, None]], axis=-1)</pre><pre>      # Performs a full training step on the collected batch.<br>      # Note: no need to mess with gradients, Keras API handles it.<br>      losses = self.model.train_on_batch(<br>        observations, [acts_and_advs, returns])</pre><pre>      logging.debug(<br>        &quot;[%d/%d] Losses: %s&quot; % (update + 1, updates, losses))</pre><pre>    return ep_rewards</pre><pre>  def <strong>_returns_advantages</strong>(<br>    self, rewards, dones, values, next_value<br>  ):<br>    # `next_value` is the bootstrap value estimate (critic).<br>    returns = np.append(<br>      np.zeros_like(rewards), next_value, axis=-1)</pre><pre>    # Returns are calculated as discounted sum of future rewards.<br>    for t in reversed(range(rewards.shape[0])):<br>      returns[t] = <br>        rewards[t] + self.gamma * returns[t + 1] * (1 - dones[t])<br>    returns = returns[:-1]</pre><pre>    # Advantages eq returns - baseline (value estimates here).<br>    advantages = returns - values</pre><pre>    return returns, advantages</pre><pre>  def <strong>test</strong>(self, env, render=False):<br>    # Unchanged from the previous section.<br>    ...</pre><pre>  def <strong>_value_loss</strong>(self, returns, value):<br>    # Unchanged from the previous section.<br>    ...</pre><pre>  def <strong>_logits_loss</strong>(self, actions_and_advantages, logits):<br>    # Unchanged from the previous section.<br>    ...</pre><h4>Results</h4><p>We are now all set to train our single-worker A2C agent on CartPole-v0! The training process should take a couple of minutes. After the training is complete, you should see an agent achieve the target 200 out of 200 score.</p><pre>rewards_history = agent.train(env)<br>print(&quot;Finished training, testing...&quot;)<br>print(&quot;%d out of 200&quot; % agent.test(env)) <strong># 200 out of 200</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*S-8mrK4jud9luAct.gif" /></figure><p>In the source code, I include some additional helpers that print out running episode rewards and losses, along with basic plotter for the rewards history.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*CPgX-Y4FRSx5HSNN.png" /></figure><h3>Static Computational Graph</h3><p>With all of this eager mode excitement, you might wonder if using a static graph is even possible anymore. Of course, it is! And it takes just one line!</p><pre>with tf.Graph().as_default():<br>  print(tf.executing_eagerly()) # False</pre><pre>  model = Model(num_actions=env.action_space.n)<br>  agent = A2CAgent(model)</pre><pre>  rewards_history = agent.train(env)<br>  print(&quot;Finished training, testing...&quot;)<br>  print(&quot;%d out of 200&quot; % agent.test(env)) <strong># 200 out of 200</strong></pre><p>There is one caveat though: during static graph execution, we can not just have Tensors laying around, which is why we needed that trick with the separate ProbabilityDistribution model definition. In fact, while I was looking for a way to execute in static mode, I discovered one interesting low-level detail about models built through the Keras API…</p><h3>One More Thing…</h3><p>Remember when I said TensorFlow runs in eager mode by default, even proving it with a code snippet? Well, I lied! Kind of.</p><p>If you use Keras API to build and manage your models, then it attempts to compile them as static graphs under the hood. So what you end up with is the performance of static graphs with the flexibility of eager execution.</p><p>You can check the status of your model via the model.run_eagerly flag. You can also force eager mode by manually setting it, though most of the times you probably don’t need to – if Keras detects that there is no way around eager mode, it backs off on its own.</p><p>To illustrate that it is running as a static graph here is a simple benchmark:</p><pre># Generate 100k observations to run benchmarks on.<br>env = gym.make(&#39;CartPole-v0&#39;)<br>obs = np.repeat(env.reset()[None, :], 100000, axis=0)</pre><h4>Eager Benchmark</h4><pre>%%time</pre><pre>model = Model(env.action_space.n)<br>model.run_eagerly = True</pre><pre>print(&quot;Eager Execution:  &quot;, tf.executing_eagerly())<br>print(&quot;Eager Keras Model:&quot;, model.run_eagerly)</pre><pre>_ = model(obs)</pre><pre>######## Results #######</pre><pre>Eager Execution:   True<br>Eager Keras Model: True<br>CPU times: user 639 ms, sys: 736 ms, total: 1.38 s</pre><h4>Static Benchmark</h4><pre>%%time</pre><pre>with tf.Graph().as_default():<br>    model = Model(env.action_space.n)</pre><pre>print(&quot;Eager Execution:  &quot;, tf.executing_eagerly())<br>    print(&quot;Eager Keras Model:&quot;, model.run_eagerly)</pre><pre>_ = model.predict(obs)</pre><pre>######## Results #######</pre><pre>Eager Execution:   False<br>Eager Keras Model: False<br>CPU times: user 793 ms, sys: 79.7 ms, total: 873 ms</pre><h4>Default Benchmark</h4><pre>%%time</pre><pre>model = Model(env.action_space.n)</pre><pre>print(&quot;Eager Execution:  &quot;, tf.executing_eagerly())<br>print(&quot;Eager Keras Model:&quot;, model.run_eagerly)</pre><pre>_ = model.predict(obs)</pre><pre>######## Results #######</pre><pre>Eager Execution:   True<br>Eager Keras Model: False<br>CPU times: user 994 ms, sys: 23.1 ms, total: 1.02 s</pre><p>As you can see, eager mode is behind static, and by default, our model was indeed executed statically, almost matching the explicitly static execution.</p><h3>Conclusion</h3><p>Hopefully, this has been an illustrative tour of both DRL and the shiny new things in TensorFlow 2.x. Note that many of the design choice discussions are <a href="https://groups.google.com/a/tensorflow.org/forum/#!forum/developers">open to the public</a>, and everything is subject to change. If there is something about TensorFlow, you especially dislike (or like :) ), let the developers know!</p><p>A lingering question people might have is if TensorFlow is better than PyTorch? Maybe. Maybe not. Both are excellent libraries, so it is hard to say one way or the other. If you are familiar with PyTorch, you probably noticed that TensorFlow 2.x has caught up and arguably avoided some of the PyTorch API pitfalls.</p><p>At the same time, I think it would be fair to say that PyTorch was affected by the design choices of TensorFlow. What is clear is that this “competition” has resulted in a net-positive outcome for both camps!</p><p><em>Originally published at </em><a href="http://inoryy.com/post/tensorflow2-deep-reinforcement-learning/"><em>inoryy.com</em></a><em> on January 20, 2019.<br>Updated on January 13, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8e62102680d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I Majored in Statistics for a Career in Artificial Intelligence]]></title>
            <link>https://medium.datadriveninvestor.com/why-i-majored-in-statistics-for-a-career-in-artificial-intelligence-3cbd09ba7ea6?source=rss-30cb90653dc4------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cbd09ba7ea6</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Roman Ring]]></dc:creator>
            <pubDate>Fri, 04 Jan 2019 22:00:00 GMT</pubDate>
            <atom:updated>2019-01-09T05:49:32.703Z</atom:updated>
            <content:encoded><![CDATA[<figure><a href="http://www.track.datadriveninvestor.com/1B9E"><img alt="" src="https://cdn-images-1.medium.com/max/700/0*envQbi2F-6I4bHaF" /></a></figure><p>Undergraduate major is often the first significant career decision a person makes in his life. As artificial intelligence (AI) becomes more and more ingrained in our society, many people begin to consider a career in AI as a viable choice in their life. However, it is still very rare to have an undergraduate degree fully dedicated to AI, so people opt for what they perceive to be the next best thing — computer science. But I believe there is a better alternative: statistics, and in this blog post I will try to explain why, based on my own example.</p><p>In recent years AI and its many subfields like machine learning (ML) have exploded in popularity and are on track to pretty much take over every industry out there. And people have noticed: the introductory course Machine Learning (CS229) at Stanford had over a thousand students enrolled in the fall of 2017!</p><blockquote><em>Stanford’s first day of class — record-breaking 1040 people already enrolled for on-campus Machine Learning (CS229). Wow! </em><a href="https://twitter.com/danboneh?ref_src=twsrc%5Etfw"><em>@danboneh</em></a></blockquote><blockquote><em>— Andrew Ng (@AndrewYNg) </em><a href="https://twitter.com/AndrewYNg/status/912382154155352064?ref_src=twsrc%5Etfw"><em>September 25, 2017</em></a></blockquote><p>Well, I want to let you in on a little secret: a majority of AI/ML, including the hip new trend you’ve probably heard about called deep learning, is just applied statistics in disguise: many ML techniques and algorithms are either fully borrowed from or heavily rely on the theory from statistics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/0*ltb7XPB_ehZK_zm6.png" /></figure><p>Unfortunately, statistics is lacking in their PR department, which causes many people to misunderstand the field. When I said I am majoring in statistics, many of my peers&#39; reactions were confusion at best, with some even assuming I was joking. In fact, I was often looked down upon by both maths and CS majors: mathematicians considered stats to be not “pure” enough, whereas CS people thought it’s not engineering-oriented enough. What’s funny is that I actually agree with both of those camps, but I believe those to be pros rather than cons. So let’s review some of the core subjects I’ve taken during my undergraduate studies and how they have helped me with AI/ML.</p><p><strong>Mathematical Analysis.</strong> You’ve probably heard of or even taken the “practical” alternative to it — Calculus, which is an okay subject, but in my opinion, by not focusing on the theory behind the various theorems and lemmas a student never actually builds an intuitive understanding. And boy does it help in AI/ML. The topics at the heart of MA — continuity and differentiability, are also what is behind most of AI/ML algorithms.</p><p><strong>Probability Theory and Statistics.</strong> Again, some version of this subject is likely taught in CS degrees as well, but theory is typically avoided. I’ve had no trouble diving head first into reinforcement learning thanks to a deep and intuitive understanding of random variables and their estimates, expectations, distributions, and so on.</p><p><strong>Numerical Methods.</strong> Speaking of what is behind most AI/ML, this subject tackles the questions of function optimization and approximation. And if function approximation sounds alien to you, then perhaps you’ve heard of its special case — artificial neural networks.</p><p><strong>Matrix Calculus.</strong> And while we’re on the subject of artificial neural networks, you’ve probably heard that they are represented as a chain of differentiable matrix operations. Well, here is a whole subject dedicated to understanding how to transfer your multivariate differentiation theory into the world of linear algebra.</p><p><strong>Monte Carlo Methods.</strong> Have you ever wondered how probability theory is applied in practice? How can your computer generate random variables from any distribution? Well, this subject covers this and much more. And if you are into reinforcement learning then this course is probably the most important one to take as it covers a large chunk of the theory behind it. For example, the REINFORCE family of algorithms are built on the Monte Carlo methods.</p><p><strong>Stochastic Processes.</strong> Speaking of theory behind RL, here’s a whole subject dedicated to dealing with probability distributions over time. Markov Chains, Renewals, Queues, Brownian Motions, Gaussian Processes, …</p><p><strong>Data Analysis.</strong> Did I mention that a majority of machine learning is actually applied statistics? This course intimately covers the theory behind what people would refer to as classical ML — from simple linear regression to generalized models.</p><p><strong>Experimental Design.</strong> As we are starting to reach the limitations of our hardware the various ML/RL experiments become increasingly expensive in terms of wall-clock time. More and more people are now looking for ways to extract similar quality of information with less effort. Well, statisticians have been working on this problem for decades and you can learn all about it in this course.</p><p>But what about programming, you might ask? Well, with a balanced curriculum you actually get quite a fair share of computer science. In fact, with a couple of good elective choices, you can cover most of the fundamental knowledge necessary to work as a software engineer if you ever wanted to switch. For example here are the CS courses I have had in my undergrad:</p><ul><li>Intro to Programming (Python)</li><li>Object Oriented Programming (Java)</li><li>Algorithms and Data Structures (Java)</li><li>Database Systems (SQL)</li><li>Operating Systems (Python / Java)</li><li>Programming Languages (Prolog, Haskell, Scala, OCaml, C)</li></ul><p>So where’s the catch? Well, the problem with having a subject that is so widely misunderstood and unpopular is that the recruiters looking at your resume might assume you’re one of those hippies that prefer pen &amp; paper to a keyboard and pass you over for a “safe” computer science guy. Unfortunately, to get around this I think it’s inevitable that you still have to get the desired stamp, either via double major or with a computer science focused masters degree.</p><p>In conclusion, I believe statistics to be the perfect major for a career in AI. As I am wrapping up my first semester of computer science masters I feel that I am often quite ahead of my peers specifically because of my undergraduate background and hopefully I have persuaded some of you to give it a shot!</p><p><em>Originally published at </em><a href="http://inoryy.com/post/why-study-statistics-for-artificial-intelligence/"><em>inoryy.com</em></a><em> on January 4, 2019.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cbd09ba7ea6" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/why-i-majored-in-statistics-for-a-career-in-artificial-intelligence-3cbd09ba7ea6">Why I Majored in Statistics for a Career in Artificial Intelligence</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compiling PyTorch 4.0 on Ubuntu 17.10 with CUDA 9.0 and Python 3.6]]></title>
            <link>https://medium.com/@Inoryy/compiling-pytorch-4-0-on-ubuntu-17-10-with-cuda-9-0-and-python-3-6-6769a0df56d5?source=rss-30cb90653dc4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6769a0df56d5</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[gcc]]></category>
            <dc:creator><![CDATA[Roman Ring]]></dc:creator>
            <pubDate>Sun, 11 Mar 2018 09:11:22 GMT</pubDate>
            <atom:updated>2018-03-11T09:18:04.895Z</atom:updated>
            <content:encoded><![CDATA[<p>Using bleeding edge software is always painful, but getting PyTorch 4.0 to work proved to be next level for me. Hopefully, I can alleviate the pain for everybody else involved.</p><p>First thing first, the only <em>g++</em> compiler that worked for me was 4.8.5, which is quite ancient and buggy (for reference, Ubuntu 17.10 comes with 7.2.0). Not only do you have to install it, you also have to make PyTorch and CUDA to pick it up as the compiler. <em>update-alternatives</em> to the rescue!</p><pre>sudo apt-get install -y gcc-4.8</pre><pre>sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 70 --slave /usr/bin/g++ g++ /usr/bin/g++-7<br>sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 48 --slave /usr/bin/g++ g++ /usr/bin/g++-4.8</pre><pre>sudo update-alternatives --set gcc /usr/bin/gcc-4.8</pre><p>As a result, your “default” gcc version is now 4.8.5:</p><pre>inoryy@inoryy-PC:~$ gcc --version<br>gcc (Ubuntu 4.8.5-4ubuntu6) 4.8.5</pre><p>But, as I mentioned, it’s an ancient compiler with some bugs “built-in”, so you’ll have manually fix them. Open</p><pre>/usr/include/x86_64-linux-gnu/c++/4.8/bits/c++config.h</pre><p>and<em> </em>add the following lines to enable C99 math headers:</p><pre>#define _GLIBCXX_USE_C99 1<br>#define _GLIBCXX_USE_C99_MATH 1</pre><p>Now you’re all set to start preparing for building PyTorch from source:</p><pre>git clone <a href="https://github.com/pytorch/pytorch">https://github.com/pytorch/pytorch</a> &amp;&amp; cd pytorch</pre><p>Well, not quite. You now have to add <em>atomic.h</em> header file. Yes, apparently a whole critical header was forgotten when <em>gcc 4.8.5</em> was shipped. Did I mention it was buggy?</p><p>There’s a bunch of them you can find online, all with slightly different content and versions. The only one that worked for me was this <a href="https://github.com/gcc-mirror/gcc/blob/0fadac62abe1f559def14f01391670980b3316b8/gcc/ginclude/stdatomic.h">specific one</a>:</p><pre>wget <a href="https://raw.githubusercontent.com/gcc-mirror/gcc/0fadac62abe1f559def14f01391670980b3316b8/gcc/ginclude/stdatomic.h">https://raw.githubusercontent.com/gcc-mirror/gcc/0fadac62abe1f559def14f01391670980b3316b8/gcc/ginclude/stdatomic.h</a> -P aten/src/TH/</pre><p>You’re now finally ready to compile PyTorch!</p><pre>python setup.py install</pre><p>Assuming it doesn’t crash mid-way, you can verify the build was succesful by checking PyTorch version (note: do it in <a href="https://github.com/pytorch/pytorch/issues/574">another directory than the sources</a>)</p><pre>inoryy@inoryy-PC:~$ python<br>Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19) <br>[GCC 7.2.0] on linux<br>Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.<br>&gt;&gt;&gt; import torch<br>&gt;&gt;&gt; torch.__version__<br>&#39;0.4.0a0+15cc24a&#39;</pre><p>And finally, don’t forget to set gcc back to 7.2.0!</p><pre>sudo update-alternatives --set gcc /usr/bin/gcc-7</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6769a0df56d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>