<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ARVI AI on Medium]]></title>
        <description><![CDATA[Stories by ARVI AI on Medium]]></description>
        <link>https://medium.com/@ai.info?source=rss-b1eb6c162d19------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FHoJvoQzBfn-ArE2pjZXqQ.jpeg</url>
            <title>Stories by ARVI AI on Medium</title>
            <link>https://medium.com/@ai.info?source=rss-b1eb6c162d19------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:55:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ai.info/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Open Alpha Zero by ARVI Lab]]></title>
            <link>https://medium.com/@ai.info/open-alpha-zero-by-arvi-lab-75b602d2f039?source=rss-b1eb6c162d19------2</link>
            <guid isPermaLink="false">https://medium.com/p/75b602d2f039</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[ARVI AI]]></dc:creator>
            <pubDate>Tue, 12 Jun 2018 10:48:13 GMT</pubDate>
            <atom:updated>2018-06-12T10:49:17.856Z</atom:updated>
            <content:encoded><![CDATA[<p>Here at <a href="http://ai.arvilab.com/">ARVI Lab</a> we are working hard on recreating state of the art in Deep Reinforcement learning. We are now ready to proudly present our own DRL training framework based on the Alpha Zero approach!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*F7uPLvj4GUh8qQ4xf6NFuw.png" /></figure><p>Train an AI to play any game with Python interface using either Alpha Go or Alpha Go Zero training process. The main goal of the project is to maximize training efficiency using regular machines and GPUs. The features of the project:</p><ul><li>Build your own neural network in Keras</li><li>Optimize your neural network for inference using one line of code</li><li>Distribute your self-play generation across multiple machines and GPUs via configurations</li><li>Distribute your neural network training across multiple machines and GPUs using Keras distributed training or Uber’s Horovod</li><li>Train your AI to play games with both complete (go, chess, checkers) and incomplete informations (card games)</li></ul><h3>Getting started</h3><p>The project has built-in support of two games:</p><ul><li>a board game <a href="https://simple.wikipedia.org/wiki/Checkers">american checkers</a>, also known as draughts;</li><li>a card game <a href="https://en.wikipedia.org/wiki/Durak">durak</a>;</li></ul><p>We strongly recommend to read the series of articles about our project:</p><ul><li><a href="https://medium.com/@ai.info/building-your-own-alpha-zero-part-1-intuition-f66023df7cef">Builing your own Alpha Zero: Introduction</a></li><li><a href="https://medium.com/@ai.info/building-your-own-alpha-zero-part-2-decisions-cb4f1b90b830">Builing your own Alpha Zero: Decision Making</a></li><li>Builing your own Alpha Zero: Training (Coming soon)</li><li>Builing your own Alpha Zero: Distributed training and optimization (Coming soon)</li></ul><p>To get started with Alpha Zero by ARVI Lab check out the <a href="https://github.com/ARVILab/open_alpha_zero">Getting started</a> section on GitHub.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75b602d2f039" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building your own Alpha Zero. Part 2: Decisions]]></title>
            <link>https://medium.com/@ai.info/building-your-own-alpha-zero-part-2-decisions-cb4f1b90b830?source=rss-b1eb6c162d19------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb4f1b90b830</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[alphazero]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[ARVI AI]]></dc:creator>
            <pubDate>Wed, 06 Jun 2018 10:02:34 GMT</pubDate>
            <atom:updated>2018-06-06T10:09:25.323Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*isnlFJwu5GRcFg8p95PJuA.png" /></figure><p><em>“I’ve looked forward in time to see all of the possible outcomes of a coming conflict.” — Doctor Stephen Vincent Strange, a superhero from Marvel Cinematic Universe.</em></p><p>If only we could predict all of the outcomes of our actions ahead of time! Unfortunately, we can’t, and neither can machines unless a task they are trying to solve is a really simple one. However, machines still can predict future states of the game a lot faster than any human. This ability enables AI to outperform humans in checkers, chess, go, and various other games. Alpha Zero utilizes the power of predictions as well using an algorithm called <strong>Monte Carlo Tree Search (MCTS)</strong>.</p><p>This algorithm lets an AI look into the most promising of the possible futures instead of just trying to imagine every possible situation. Alpha Zero combines MCTS with a deep neural network capable of evaluation of a game state by which it can then determine these most promising futures. Given enough training data, this approach gives Alpha Zero the ability to efficiently find a best possible way to victory from any game state.</p><p>The best way to learn how the MCTS works is to implement it. But, before we start our implementations, let’s do some basic coding first.</p><h3>Games and Agent</h3><p>We are going to define two simple interfaces in Python: a <strong>Game</strong> and an <strong>Agent</strong>. The <strong>Game</strong> interface lets us create a game session, get observations of a current state, take actions, and get the result of a game session. The <strong>Agent</strong> interface lets us predict a policy and a value from a current game state.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8dd0cdd2df0ed43bd595f3b577a23a2d/href">https://medium.com/media/8dd0cdd2df0ed43bd595f3b577a23a2d/href</a></iframe><p>Take a minute or two to get familiar with these interfaces. When you are ready, we are going right into the implementation of the MCTS!</p><h3>MCTS-based Agent</h3><p>Now we are going to make an MCTS-based <strong>Agent</strong>. I know, it’s weird to implement an algorithm without explanations, but trust me on this one; it’s not as hard as it looks. Reading the implementation will give you a basic idea of how the algorithm works, and it will all make sense when I explain it all in a more formal way.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/874790549a9f193e78efdff7435be33c/href">https://medium.com/media/874790549a9f193e78efdff7435be33c/href</a></iframe><p>Woah! That was a lot of code to process. It’s about time to explain what is going on here.</p><h3>Monte Carlo Tree Search</h3><p>MCTS is a policy search algorithm that balances exploration with exploitation to output an improved policy after a number of simulations of the game. MCTS builds a tree where nodes are different observations, and a directed edge exists between two nodes if a valid action can cause the state to transition from one node to another.</p><p>For each edge, we maintain a <strong>Q</strong> value denoted by <strong>Q(s, a)</strong> which is the expected reward for taking that action and <strong>N(s, a)</strong> which represents the number of times we took action a from state <strong>s</strong> across different simulations.</p><p>The hyperparameters of MCTS are:</p><ul><li><strong>number of simulations </strong>is the parameter which represents how many previously unexplored nodes we want to visit every time we call the <em>agent.predict</em> function;</li><li><strong>cpuct </strong>is the parameter controlling the degree of exploration;</li><li><strong>exploration rate </strong>is the parameter, controlling the distribution of the final policy. Setting it to a high value gives an almost uniform distribution while setting it to 0 makes us always select the best action.</li></ul><p>The process of exploration is iterative. It starts with getting an <strong>observation </strong>from the game and getting an<strong> observation hash </strong>using <strong>the </strong>method:</p><p><em>game.get_observation_str(observation):</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a31a35f1a7dd6b8f2d2b03f81e1d2f4/href">https://medium.com/media/7a31a35f1a7dd6b8f2d2b03f81e1d2f4/href</a></iframe><p>We iterate over a <strong>number of simulations</strong>, each time performing exploration of our tree until we find a previously unexplored or a terminal node using the <em>search</em> method. Note that we should pass a <strong>copy</strong> of a current game state to the <em>search</em> method, as it changes the state of the game during the exploration process.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/334944d064751d6a6236acb0d0fab779/href">https://medium.com/media/334944d064751d6a6236acb0d0fab779/href</a></iframe><p>So, the search begins…</p><p>If we reached a terminal node, we just need to propagate a value for the current player:<br>So, the search begins…</p><p>If we reached a terminal node, we just need to propagate a value for the current player:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9b3eb5e5a2d0944d8442ccd7f5f1e1c/href">https://medium.com/media/c9b3eb5e5a2d0944d8442ccd7f5f1e1c/href</a></iframe><p>If we reached a previously unexplored node, we get <strong>P(s)</strong> which is the prior probability of taking a particular action from state <strong>s</strong>according to the policy returned by our neural network. Note that the neural network <strong>Agent </strong>uses the same interface <strong>Agent</strong> we defined earlier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43fdcc9ee59564913c0caa658c743de/href">https://medium.com/media/c43fdcc9ee59564913c0caa658c743de/href</a></iframe><p>If we encountered a known state, we calculate <strong>U(s, a)</strong> values for every edge (action-&gt;state transition), which is an upper confidence bound on the <strong>Q</strong> value of our edge. These values are calculated as:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54cb0581c249d946976cf8b0b5dc11f4/href">https://medium.com/media/54cb0581c249d946976cf8b0b5dc11f4/href</a></iframe><p>Once we finished our MCTS simulations, <strong>N(s, a)</strong> values provide a good approximation for the policy from each state. The only thing left is to apply our <strong>exploration rate</strong> parameter to control the distribution of the policy values:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5137951a781766cc8d7fb4fa51e8db0/href">https://medium.com/media/f5137951a781766cc8d7fb4fa51e8db0/href</a></iframe><p>Congratulations! You now understand now how the Monte Carlo Tree Search works! However, there is one more thing left, before we’ll be able to make it work.</p><h3>Neural Network</h3><p>Do you remember these lines from our AgentMCTS?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2dc298fb3c5cf26b822a4995d02b9d71/href">https://medium.com/media/2dc298fb3c5cf26b822a4995d02b9d71/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24792e043c60c2b88e76b5e882e02560/href">https://medium.com/media/24792e043c60c2b88e76b5e882e02560/href</a></iframe><p>That is exactly how we are going to use our neural network here. It produces policy and value, so the <strong>Agent</strong> interface may be reused with the neural network too. Let’s write a neural-network based <strong>Agent</strong>! We’ll use the Keras library to make our code as clean as possible. First, we define our abstract neural network:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/332caa36f74a035cff60ac67f7e3314d/href">https://medium.com/media/332caa36f74a035cff60ac67f7e3314d/href</a></iframe><p>And now we define our <strong>Agent</strong> which will use the neural network:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c9bb7b2b1f3e64c86d616ce2d5bf463/href">https://medium.com/media/9c9bb7b2b1f3e64c86d616ce2d5bf463/href</a></iframe><p>In case you are wondering how to make neural network output a policy and a value, here is an example of a simple, convolutional neural net for Alpha Zero:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8fc6bf937ccee52432dbcad356f6302/href">https://medium.com/media/f8fc6bf937ccee52432dbcad356f6302/href</a></iframe><h3>Bringing it all together</h3><p>Today we’ve defined our core interfaces: Game, Agent and NNet. We implemented a neural-network based Agent and a Monte Carlo Tree Search — based Agent! So, to initialize our Monte Carlo Agent we just need to write the code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4230ffaf97e3747f82e6d789934c1405/href">https://medium.com/media/4230ffaf97e3747f82e6d789934c1405/href</a></iframe><p>And our agent_mcts might predict the policy and value of a game state like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e7fe8ed9e885fff8bbd95abcfcf226a/href">https://medium.com/media/9e7fe8ed9e885fff8bbd95abcfcf226a/href</a></iframe><h3>What’s next?</h3><p>Awesome! We now have our core classes almost ready for writing an Alpha Zero training pipeline. Next time we are going to show you how to train an agent to play English draughts using Monte Carlo Tree Search and a deep residual network. The game has complexity of roughly <strong>500,995,484,682,338,672,639</strong> possible positions, so the task is going to be a fun one.</p><p>Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb4f1b90b830" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building your own Alpha Zero. Part 1: Intuition]]></title>
            <link>https://medium.com/@ai.info/building-your-own-alpha-zero-part-1-intuition-f66023df7cef?source=rss-b1eb6c162d19------2</link>
            <guid isPermaLink="false">https://medium.com/p/f66023df7cef</guid>
            <category><![CDATA[alphazero]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[ARVI AI]]></dc:creator>
            <pubDate>Tue, 29 May 2018 09:07:14 GMT</pubDate>
            <atom:updated>2018-06-06T10:05:46.138Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*c0YtTx92Om4nFlWBcTe7hw.png" /></figure><p><em>“Fine, I’ll do it myself” — Thanos, a supervillain from the Marvel Cinematic Universe.</em></p><p>In human society it’s considered natural to learn from the people who already have the knowledge you seek. In other words, to be the best, you must learn from the best. However, the recent Google gaming AI does not rely on any existing knowledge and prefers to discover every possible bit of knowledge in a game by itself.</p><p>As we all know, this strategy has led to a massive success. Last year, Google published a paper describing the next generation of their gaming agents, the famous <em>Alpha Zero</em>, which was able to reach a super human level of play in go, chess and shogi. And it used only the knowledge that it generated via a lot of iterations of self-play!</p><h3><strong>Introduction</strong></h3><p>Here at <a href="http://ai.arvilab.com/">ARVI Lab</a> we are building our own team of data scientists and mathematicians capable of applying recent breakthroughs in the ﬁeld of AI to real-world problems. Unlike Alpha Zero, we are just mere humans, so we decided to learn from the best experts in our Reinforcement Learning research. In order to learn a state-of-the-art in the ﬁeld, we implemented Alpha Zero using our own resources and we are going to share our code, and experience in a series of articles.</p><p>So, without further boring introductions, let’s begin our journey with the Part One: Intuition!</p><h3>Learning to play a game without any existing knowledge</h3><p><em>“Live. Die. Repeat.”</em></p><p>Imagine yourself learning a competitive game, when no one has ever bothered to tell you the rules. What is even sadder, no one wants to play with you, so you don’t even have a partner to train with. All you have ever been taught is the set of valid <em>actions </em>you might take, and you also get a notiﬁcation the moment you lose or win a game session. Also, you can get an <em>observation </em>of the game in the form of an image on every time step of the game.</p><p>Your task is simple: you need to become the best player in the history of the game.</p><p>However, you have one useful trick up your sleeve. You can clone yourself, producing an exact same copy.</p><p>So, you begin training. First of all, you clone yourself because you need a sparring partner. When you play with the clone, you simply pick a valid action every time you have to. Every time you take an action, you record the current state of the game and the action you took. At some point, you ﬁnally lose or win.</p><p>When you look into your recordings, you try to ﬁgure out what actions were the right ones. You also try to understand why the last state of the game was terminal and how to avoid it, simultaneously trying to push your opponent into his terminal state. Once you feel like you are ready to give it another shot, you clone yourself one more time and play again.</p><p>Every time you learn from a new game session, you become a little bit better. At some point you stop just picking random actions and start using strategies. However, you still take random actions sometimes to see if there are any undiscovered opportunities for you to exploit. Eventually, you learn how to overcome the strategies you’ve developed on previous iterations.</p><p>You repeat the process a few million times. When you are done, no one can defeat you in the game.</p><p>Congratulations! You understand now how the Alpha Zero learns!</p><h3>The core components</h3><p><a href="https://medium.com/applied-data-science">There is a great Alpha Zero cheat sheet published in Applied Data Science blog.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*dAhwz9ZGboetdYXfFBV4hg.png" /></figure><p>Does it still look a bit overwhelming? Don’t worry, we’ve got you covered! Let’s break it down in a simple manner.</p><p>At the core of an AI <strong>agent</strong>, based on the Alpha Zero, there are two key components:</p><ul><li>A <strong>neural network</strong>, capable of generating a <strong>stochastic policy </strong>and a <strong>continuous value </strong>of a game state given an <strong>observation </strong>of the state;</li><li>A <strong>Monte Carlo Tree Search </strong>implementation which helps to improve the <strong>policy </strong>of the <strong>agent </strong>via simulations of future states of the game</li></ul><p>An <strong>environment </strong>(game) which the <strong>agent </strong>is trying to solve should provide:</p><ul><li>A vector of <strong>possible actions </strong>where every action is deﬁned by an index;</li><li>A vector of <strong>valid actions </strong>on every time step. The list has similar length as the vector of all actions, and every possible action is encoded as <strong>1 </strong>if it’s available or <strong>0 </strong>otherwise;</li><li>An <strong>observation </strong>of the game state from the point of view of the player which should take his action next. The observation might be encoded as a vector or a matrix;</li><li>A notiﬁcation if the <strong>terminal state </strong>of the game is reached and an index of the player who won the game;</li></ul><p>To put it simply, the stochastic <strong>policy </strong>is a vector of a similar length as a vector of possible actions, but every element of the vector represents a probability of an action. The bigger the probability, the better the action from the neural network’s point of view. The continuous <strong>value </strong>of a game state is a number between [-1, 1] which represents the likelihood of the given player winning the game from the current state.</p><p>The <strong>Monte Carlo Tree Search </strong>(MCTS) algorithm serves as an “imagination” of the <strong>agent</strong>. MCTS tries to predict what is going to happen next by using the neural network to predict next moves of the opponent.</p><h3>Bringing it all together</h3><p>First, we initialize our neural network with random values. The <strong>inputs </strong>of the neural network are of the same shape as an <strong>observation </strong>of the game, and the <strong>outputs </strong>are a policy vector and a value of the state, given the observation.</p><p>On every game step we get an <strong>observation </strong>from the game <strong>environment </strong>and feed it to the <strong>Monte Carlo Tree Search </strong>(MCTS). The MCTS algorithm uses the <strong>neural network </strong>to simulate the next <strong>number of MCTS simulations </strong>steps of the game and returns us an updated <strong>policy</strong>. We pick an action from the policy (the policy is a distribution of actions probabilities) and move to the next game step. On every game step, we collect a pair made up of a <strong>policy </strong>and an <strong>observation</strong>.</p><p>When the game is ﬁnished, we update our collected pairs by appending a game result to them. All of the pairs of the winner get <strong>1</strong>, and all of the pairs of the loser get -<strong>1</strong>.</p><p>When we feed a list of trios (<strong>policy</strong>, <strong>observation</strong>, <strong>result</strong>) to the neural network and train it, the training set is built from observations and the labels set is built from policies and results.</p><p>By repeating the process, we increase the quality of our neural network’s estimations which leads to better MCTS estimations. The better MCTS estimations we get, the better strategies our agent learns.</p><h3>What’s next?</h3><p>The idea behind Alpha Zero is pretty simple. However, the process of it’s implementation presents a lot of challenges, tricks and bottlenecks which a developer must overcome in order to train Alpha Zero efficiently.</p><p>In our next article we will dive deeper into neural network design and Monte Carlo Tree Search in order to uncover and solve these challenges. Stay tuned!</p><h4>More where this came from</h4><p>This story is published in <a href="http://blog.usejournal.com/">Noteworthy</a>, where thousands come every day to learn about the people &amp; ideas shaping the products we love.</p><p>Follow our publication to see more product &amp; design stories featured by the <a href="https://usejournal.com/?/utm_source=usejournal.com&amp;utm_medium=blog&amp;utm_campaign=guest_post">Journal</a> team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f66023df7cef" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>