<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anton Osika on Medium]]></title>
        <description><![CDATA[Stories by Anton Osika on Medium]]></description>
        <link>https://medium.com/@osikaanton?source=rss-7a1a950316ee------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RwFKWQV-YeIiW3OZvTWy-w.jpeg</url>
            <title>Stories by Anton Osika on Medium</title>
            <link>https://medium.com/@osikaanton?source=rss-7a1a950316ee------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:39:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@osikaanton/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Go-Explore solves hard-exploration problems]]></title>
            <link>https://medium.com/@osikaanton/how-go-explore-solves-hard-exploration-problems-5349b9b1bc90?source=rss-7a1a950316ee------2</link>
            <guid isPermaLink="false">https://medium.com/p/5349b9b1bc90</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[planning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Anton Osika]]></dc:creator>
            <pubDate>Sun, 28 Jul 2019 11:01:55 GMT</pubDate>
            <atom:updated>2019-07-30T20:36:44.801Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qBHgMZsDPgcR9NyKWYyWtg.jpeg" /><figcaption>Montezuma’s Revenge is an example of an Atari game that was not possible to solve with conventional Reinforcement Learning algorithms</figcaption></figure><p><a href="https://arxiv.org/pdf/1901.10995.pdf">Go-Explore</a> is an algorithm for solving <strong>hard-exploration reinforcement learning problems</strong>. It is impressive in its simplicity, and the core part of it <strong>does not require any machine learning</strong>. Go-Explore was invented at Uber and was used to beat the state-of-the-art at<strong> </strong>Montezuma’s Revenge<strong> </strong>with &gt; 3x higher score.</p><p>The idea behind Go-Explore is to keep track of states visited in previous game-episodes and to do exploration starting directly from those states, rather than resetting the environment and starting each episode from scratch.</p><h3>Explanation in 100 words</h3><p>Go-Explore boils down to the following steps:</p><ol><li>Work with low dimensional representations of the state (e.g. a downsampled images in the game screen)</li><li>Keep track of the best way to reach each state</li><li>Use a heuristic function to select previously visited states to start from (typically based on the cumulative reward, visit counts, etc.)</li><li>After selecting a state, do a rollout of ~100 random actions to collect new states</li><li>If the environment is stochastic, train a neural network policy that learns to reach previously discovered states</li></ol><p>The last step of the algorithm is necessary because if there is randomness, repeating a sequence of actions will typically not bring you to the same state that you reached the first time you followed the sequence.</p><h3>Real-world applications</h3><p>The inventors of the algorithm argue that this simple approach can be used in many difficult RL settings.</p><p>In the Go-Explore paper, it is pointed out that if the representation of the state is based on <strong>domain knowledge</strong> rather than just pure pixel values, the algorithm <strong>performs 10x better</strong>. Using domain knowledge is typically not a problem in real-world applications since manual feature extraction is often the norm.</p><p>Something that is not heavily emphasized in the Go-Explore paper is the importance of the heuristic function. It seems like <strong>tuning this function is important to make it work</strong>. In the final version they used, the function includes some hyperparameters that were selected separately for each game by doing a grid search.</p><p><strong>References</strong></p><p>[Ecoffet 2019] <a href="https://arxiv.org/pdf/1901.10995.pdf">https://arxiv.org/pdf/1901.10995.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5349b9b1bc90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is the default outcome of AI doom?]]></title>
            <link>https://medium.com/@osikaanton/is-the-default-outcome-of-ai-doom-9545de64cd94?source=rss-7a1a950316ee------2</link>
            <guid isPermaLink="false">https://medium.com/p/9545de64cd94</guid>
            <category><![CDATA[ai-safety]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[futurism]]></category>
            <category><![CDATA[future]]></category>
            <dc:creator><![CDATA[Anton Osika]]></dc:creator>
            <pubDate>Wed, 04 Jul 2018 17:42:16 GMT</pubDate>
            <atom:updated>2018-08-31T16:13:22.901Z</atom:updated>
            <content:encoded><![CDATA[<h3>How likely is it that a transition to a world with superhuman Artificial Intelligence would become a dystopia? Below are some arguments using the perspective that lifeforms subject to evolution can be either in a stable or unstable state. In summary, this perspective can help humans today when making important decisions (independent of AI).</h3><p>Let’s consider the state of our society. It consists of humans that have evolved, through survival of the fittest selection, to a transcended state where we do not simply aim to make our genes spread as widely as possible but value more enlightened things.</p><p>It is, however, critical to note that this transcended state is unstable. Unstable meaning that a small perturbation would cause the state to change significantly (and tend towards a more stable point).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eu_9bdUzp4ScrVIbzGvuqw.jpeg" /></figure><p>As an example, consider what would happen if a subgroup of the human population suddenly had a very strong, and heritable, desire to get as many offsprings as possible. This subgroup would purposefully acquire economical resources, grow and “outcompete” other humans (bribing them for raising their kids and whatnot) until every human alive was a descendant of this subgroup.</p><p>The same scenario could play out if a superintelligent AI, with the same goal, was ever born.</p><p>Taking this one step further, the only stable points of a society are those where its lifeforms act optimally in converting resources (free energy) into more copies of themselves. ¹ Furthermore, any system with enough randomness will move from its unstable to its stable points.</p><p>A stable-point society as described above does not have to be dystopian though. Its lifeforms would probably have a rich and interesting culture of exploring new ways to create copies of themselves for example. But, they would probably not be lifeforms with brains made of biological neurons!</p><p>Hence, as time passes on, the “default scenario is indeed that non-human intelligence takes over our universe from present day humans with a high likelihood. If this intelligence should be called “Artificial” is not clear, and neither that the scenario would be “doom”.</p><p>In my personal opinion, this default scenario would not necessarily be bad. But having an indifferent attitude to the scenario of moving towards a stable point would not be wise! This is because there are many cases of such a scenario that no one would argue to be desirable, and we simply do not know.</p><p>To sum it up:</p><ul><li>Our society is at an unstable point.</li><li>Compared to most cosmic alternatives our society is great.</li><li>We should take exceptional care in keeping society at an unstable point — because the alternative could play out extremely bad.</li></ul><p>So let’s not be blind towards the joy of creating more intelligence but keep in mind that small decisions can change our society towards an extremely medieval and competitive one.</p><p><em>I hope you appreciated this post. <br>Please comment or clap to let me know if this was interesting or not!</em></p><p><em>This post did not go in to the possibility of the Malthusian trap — an equilibria where people reproduce until the limit of starvation — and why (/why not) this is the stable point for intelligent lifeforms and other important topics. These are left that in the references.</em></p><p><em>Footnotes and references:</em></p><p>¹ This holds true because if a societys lifeforms act sub-optimally, a random perturbation on one lifeform could spread and outcompete all other lifeforms, and changing the system drastically. Which is the defintion of an unstable point.</p><p><em>Bostrom, Nick. Superintelligence. Dunod, 2017.</em></p><p><a href="https://en.wikipedia.org/wiki/An_Essay_on_the_Principle_of_Population">https://en.wikipedia.org/wiki/An_Essay_on_the_Principle_of_Population</a></p><p><a href="https://en.wikipedia.org/wiki/Malthusian_trap">https://en.wikipedia.org/wiki/Malthusian_trap</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9545de64cd94" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>