<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jan Forster on Medium]]></title>
        <description><![CDATA[Stories by Jan Forster on Medium]]></description>
        <link>https://medium.com/@8B_EC?source=rss-948bd4c54478------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ff2mrWiF3LnSW3-_.jpg</url>
            <title>Stories by Jan Forster on Medium</title>
            <link>https://medium.com/@8B_EC?source=rss-948bd4c54478------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:03:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@8B_EC/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tutorial: Serving Machine Learning Models with FastAPI in Python]]></title>
            <link>https://medium.com/@8B_EC/tutorial-serving-machine-learning-models-with-fastapi-in-python-c1a27319c459?source=rss-948bd4c54478------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1a27319c459</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fastapi]]></category>
            <dc:creator><![CDATA[Jan Forster]]></dc:creator>
            <pubDate>Sat, 15 Feb 2020 22:02:15 GMT</pubDate>
            <atom:updated>2020-02-15T22:02:15.441Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdNF54AgJqblUP_W57fR1w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@k8townsend?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Kate Townsend</a> on <a href="https://unsplash.com/s/photos/serving?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this tutorial I’ll show how to use FastAPI to quickly and easily deploy and serve machine learning models in Python as a RESTful API.</p><p><em>The code covered in this tutorial can be found here: </em><a href="https://github.com/eightBEC/fastapi-ml-skeleton"><em>https://github.com/eightBEC/fastapi-ml-skeleton</em></a></p><h3>Scenario</h3><p>Imagine that you’ve just trained a new regression model with a nice data set to predict house prices using a little Python script. Now you want to enable others to use the model easily or offer it as an API. Often the question arises by which framework the REST API will be provided — as there are many to choose from: build one yourself from scratch using the Requests library, use Django or Flask and so on — just to name a few. Your API should be fast, reliable, easy and fast to develop and ready to be used in production.</p><p>Let’s have a look at FastAPI.</p><h3>What’s FastAPI?</h3><p>FastAPI is a framework created by Sebastián Ramírez for building APIs using Python ≥ 3.6. It’s super fast, easy and quick to learn and implement, production-ready. If you want to learn about all the great features, have a look at the <a href="https://fastapi.tiangolo.com/features/">documentation</a>. After going through the tutorial you’ll probably understand why so many people and companies use and like it.</p><p><a href="https://github.com/tiangolo/fastapi">GitHub - fastapi/fastapi: FastAPI framework, high performance, easy to learn, fast to code, ready for production</a></p><h3>Walkthrough</h3><h4>Structure</h4><p>The repository is organized based on best practices from various sources (e.g. <a href="https://docs.python-guide.org/#">The Hitchhiker’s Guide to Python</a> and the <a href="https://github.com/tiangolo/full-stack-fastapi-couchbase">FastAPI full stack app</a>). It contains the following resources. I’ve tried to structure the code as intuitive as possible, but you’ll find a commented tree of the project structure below:</p><pre>.<br>├── LICENSE<br>├── MANIFEST<br>├── README.md<br>├── docs<br>│   ├── DOCS.md<br>│   ├── authorize.png<br>│   ├── sample_payload.json     # Sample Payload to test ML model<br>│   └── sample_payload.png<br>├── fastapi_skeleton            # Skeleton Module<br>│   ├── __init__.py.             <br>│   ├── api                     # API related code<br>│   │   ├── __init__.py<br>│   │   └── routes              # All routes provided by API<br>│   │       ├── __init__.py<br>│   │       ├── heartbeat.py    # Route to check if is server is up<br>│   │       ├── prediction.py   # Route to predict using ML model<br>│   │       └── router.py       # Main router to  serves the routers<br>│   ├── core<br>│   │   ├── __init__.py<br>│   │   ├── config.py           # Server configuration helper<br>│   │   ├── event_handlers.py   # Handle server start/stop<br>│   │   ├── messages.py         # Shared messages/resources<br>│   │   └── security.py         # Common security helpers<br>│   ├── main.py                 # App entrypoint<br>│   ├── models<br>│   │   ├── __init__.py<br>│   │   ├── heartbeat.py        # Data model for heartbeat response<br>│   │   ├── payload.py          # Data model for ML model payload<br>│   │   └── prediction.py       # Data model for prediction result<br>│   └── services<br>│       ├── __init__.py      <br>│       └── models.py           # Services to provide ML model<br>├── requirements.txt            # Project requirements<br>├── sample_model                # Sample ML model folder<br>│   ├── lin_reg_california_housing_model.joblib # Sample ML model<br>│   └── model_description.md    # Description of ML model params                  <br>├── setup.py                    # Python setup script for dist<br>├── tests                       # Test folder<br>│   ├── __init__.py<br>│   ├── conftest.py             # Test configuration / bootstrap<br>│   ├── test_api                # API tests<br>│   │   ├── __init__.py <br>│   │   ├── test_api_auth.py    # API authentication test cases<br>│   │   ├── test_heartbeat.py   # API Heartbeat test cases<br>│   │   └── test_prediction.py  # API ML model prediction test cases<br>│   └── test_service<br>│       ├── __init__.py       <br>│       └── test_models.py      # ML model test cases<br>└── tox.ini                     # Tox configuration</pre><h4>Install</h4><p>Use your preferred virtual Python env (e.g. virtualenv or conda) and the required packages in your local environment with:</p><pre>pip install -r requirements</pre><h4>Setup</h4><ol><li>Duplicate the .env.example file and rename it to .env</li><li>In the .env file configure the API_KEY entry. The key is used for authenticating our API. <br>A sample API key can be generated using Python REPL:</li></ol><pre>import uuid<br>print(str(uuid.uuid4()))</pre><h4>Run It</h4><ol><li>Start your app with:</li></ol><pre>uvicorn fastapi_skeleton.main:app</pre><p>2. Go to <a href="http://localhost:8000/docs">http://localhost:8000/docs</a>.</p><p>3. Click Authorize and enter the API key as created in the Setup step.</p><p>4. You can use the sample payload from the docs/sample_payload.json file when trying out the house price prediction model using the API.</p><h3>Summary</h3><p>In this tutorial we looked at how easy it is to deploy machine learning models using FastAPI. Feel free to reuse the skeleton code provided to get started. It’s fully tested and can be easily extended.</p><h3>Next:</h3><p>If you’re familiar with container technologies, you have probably missed the Dockerfile in this tutorial. No worries, in the next post, I’m going to refine the skeleton code so that it can be easily deployed in a containerised environment and eventually used in Kubernetes.</p><p>I hope this tutorial and code helps you to get started using the great FastAPI library!</p><p><em>If you enjoyed this post, </em><strong><em>please click </em>👏<em> </em></strong><em>so that others can read it as well. Follow me on Twitter </em><a href="https://twitter.com/8B_EC"><em>@8B_EC</em></a><em> for the latest updates or just to say hi :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1a27319c459" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NLP in Practice: Building WKS Type Systems with MURAL]]></title>
            <link>https://medium.com/@8B_EC/nlp-in-practice-building-wks-type-systems-with-mural-e82326525ccc?source=rss-948bd4c54478------2</link>
            <guid isPermaLink="false">https://medium.com/p/e82326525ccc</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[watson-knowledge-studio]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Jan Forster]]></dc:creator>
            <pubDate>Mon, 02 Dec 2019 16:52:52 GMT</pubDate>
            <atom:updated>2019-12-02T16:52:52.121Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9n83J8hoMbypKf4wfFW3Sg.png" /><figcaption>Original Photo by <a href="https://pixabay.com/users/KlausHausmann-1332067/">Klaus Hausmann</a> on <a href="https://pixabay.com/photos/stone-age-painting-mural-lascaux-2115390/">Pixabay</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*W-3ZZ0cC0qo4mmuW_dZFvg.png" /></figure><p>Co-Authored by <a href="https://medium.com/u/d635f5ef4811">Christina Niegel</a> and <a href="https://medium.com/u/948bd4c54478">Jan Forster</a></p><p>One of the IBM Watson services that allow a very deep understanding of complex natural language is <a href="https://www.ibm.com/cloud/watson-knowledge-studio">Watson Knowledge Studio</a> (WKS). It helps users to design and develop Machine Learning annotators trained on understanding a domain language, not only regarding specific terminology but also underlying related information. Accessing not only strings and keywords, but context-sensitive information can unlock the semantic meaning of terms and phrases and helps to concentrate on what’s relevant for the goal of the NLP application.</p><h3><strong>A Type System?</strong></h3><p>A Type System is the abstract representation of information that is relevant within the domain and will be the guideline for the “human annotators”. They are going to train the system and help WKS to learn what to do with text.</p><p>For all those who already developed annotators with WKS, it’s nothing new when we say: a well-designed Type System can decide about the success or fail of your WKS project. Creating a Type System is often a challenging task and underestimating it can turn out in non-scalable solutions and unsatisfied users of the end product.</p><p>Unfortunately, contrary to common beliefs, we have to admit that there is no recipe for a universal Type System — sorry. The number of different Type Systems varies as much as there are business problems and domain languages. But even if there’s not a single solution that solves every problem, we’d like to share our approach that can ease the process of building a feasible Type System.</p><h3><strong>Use Case Example</strong></h3><p>Imagine yourself being in a situation where your client, a large smartphone retailer, wants to allow his customers to find smartphones with their natural language using a chatbot. People might ask the following questions to find a suitable smartphone:</p><blockquote>“Which iPhone can I get for less than 500 dollars?”<br>“Which phone has more than 4GB of RAM?”<br>“How many pixels does the Pixel have?”</blockquote><p>Your client needs to extract the relevant information from the customers’ questions to provide only smartphone suggestions that meet their criteria.</p><h3><strong>Goals of this Blog Entry</strong></h3><p>Given this use case, we want to share our experiences of the domain adaptation process:</p><ol><li>Share our method to build a robust Type System that is focused on the business requirements and outcome. Questions to ask yourself: What are the entities and relations I need to recognize and extract to achieve the business goal?</li><li>Support you in abstracting away from a specific example towards a scalable Type System design process.</li><li>Underline the importance of the mantra: “Consistency is key”.</li></ol><h3><strong>WKS Terminology</strong></h3><p>Before we get started, let’s make sure we are using the same terms in the form of a “mini glossary”. More info can be found in the official <a href="https://cloud.ibm.com/docs/services/watson-knowledge-studio?topic=watson-knowledge-studio-typesystem">WKS documentation</a>.</p><h4>Type System</h4><p>The Type System defines elements you want to label using annotations. It defines how content can be annotated using entities and relations.</p><h4>Mention</h4><p>A mention is a span of text that you consider relevant in your text.</p><h4>Entity Type</h4><p>An Entity Type is a category you want to be able to recognize in your text to solve your business problem. A mention in the text that belongs to a particular Entity Type is called Entity Mention.</p><h4>Entity Subtype</h4><p>Entity Subtypes allow the further refinement of Entity Types.</p><h4>Relation Type</h4><p>A Relation Type defines a directed relationship between two entities within a single sentence.</p><h3><strong>What is MURAL?</strong></h3><p>No, we’re not traveling back in time to the Stone Age. MURAL is an online collaboration whiteboard platform. When we met in client engagement in 2018 after working on various NLP projects involving Watson Knowledge Studio and <a href="https://www.ibm.com/cloud/watson-discovery">Watson Discovery Service</a>, we came up with the idea to try out MURAL for our domain adaptation process. We both liked using it for Design Thinking sessions before, so we gave it a try.</p><h3><strong>The Flow</strong></h3><h4><strong>Corpus Studies</strong></h4><p>Before you define entities or relations, you should always read through the domain corpus and other existing language artifacts while having the business problem in mind. This takes time and is a tedious task, but it is necessary. Free your mind from any “common” Entity Types that you might have already in mind just because you know that domain. This can be misleading and brings a bias to your Type System right from the beginning.</p><h4><strong>Mention Collection</strong></h4><p>Post the relevant mentions that occur during your “corpus studies” to your MURAL board.</p><p><em>In our example, this could be terms and phrases collected from end-users that want to find a new smartphone.</em></p><figure><img alt="An overview of mentions in domain." src="https://cdn-images-1.medium.com/max/1024/1*EbsbWUI6Jmb3AARQ2q7fLw.png" /></figure><h4><strong>Entity Type Definition</strong></h4><p>When you have covered a representative set of mentions on your board, start to cluster and align the mentions: try to give those clusters names. These would be the Entity Type names.</p><p><em>Referring to our initial example user question “</em>Which iPhone can I get for less than 500 dollars?<em>” you can see the relevant Entity Type clusters in our Type System.</em></p><figure><img alt="An overview of entity types of a domain." src="https://cdn-images-1.medium.com/max/1024/1*sDy4PutqmZxmPQIKCN_G9g.png" /></figure><h4><strong>Relation Type Definition</strong></h4><p>Sketch possible relations between the entity clusters. Here it is important to keep the business goal and the value of the product in mind. The more complex your Type System gets, the more difficult it will be to train and maintain. You don’t need to build a domain ontology. Focus on relations that help to solve your business problem.</p><p><em>In the image below you can see the possible relations between the relevant entities we defined for our use case.</em></p><figure><img alt="An overview of relations of different entity types within a domain" src="https://cdn-images-1.medium.com/max/1024/1*VRn7UrqObGYVveT_q1RHQg.png" /></figure><h4><strong>Transfer to WKS</strong></h4><p>When you are ready, transfer the Entity Types and Relation Types into WKS and start annotating some documents. It’s completely OK if you run into problems regarding ambiguous mentions, unclear lengths of spans (multi-token Entity Mentions) or missing relations in the first iteration. Usually, you refine your Type System and go through the above steps again.</p><figure><img alt="Mention view in Watson Knowledge Studio" src="https://cdn-images-1.medium.com/max/1024/1*_IBQzBSQYuwikCghwRGiKg.png" /><figcaption>Mention view in Watson Knowledge Studio</figcaption></figure><figure><img alt="Relation view in Watson Knowledge Studio" src="https://cdn-images-1.medium.com/max/1024/1*Ue0_y_o2ZTCy6WPaAiJjDA.png" /><figcaption>Relation view in Watson Knowledge Studio</figcaption></figure><h4><strong>Cheat Sheet</strong></h4><p>Critical mentions, that might be confusing for the team of human annotators, can be collected on a separate MURAL mention board. This board serves as a cheat sheet. It’s also a great way to collaborate and document changes and extensions to your Type System. It will save time when syncing, it is easier to lookup than checking dictionaries in WKS or Excel and ensures consistency within your team and your annotations. Avoid duplicates or overlapping Entity Mentions across Entity Type clusters.</p><p><strong><em>Pro Tip ;)</em></strong><br>You can use icons in MURAL to highlight difficult or ambiguous examples that human annotators need to know about.</p><figure><img alt="A cheat sheet for mentions of entity types." src="https://cdn-images-1.medium.com/max/1024/1*HH-DJ4jyCY2ciSIRtMongg.png" /></figure><h3><strong>That’s it</strong></h3><p>We hope you enjoyed this kind of hands-on session creating a WKS Type System and hope it was fun to follow the use case.</p><p>If you want to build your own Type System in <a href="https://www.ibm.com/cloud/watson-knowledge-studio">WKS</a>, get started on the <a href="https://ibm.biz/BdjViX">IBM Cloud for free</a> today!</p><h3>About Us</h3><p><a href="https://medium.com/u/d635f5ef4811">Christina Niegel</a> and <a href="https://medium.com/u/948bd4c54478">Jan Forster</a> are NLP practitioners in the IBM Watson team in Europe. We help our clients to use the full potential of our Watson products and solutions.</p><p>We hope this content was helpful to you! Please provide feedback in the comments below. If you have any questions, let us know, we’re happy to help.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e82326525ccc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reinforcement Learning: Bootstrapping Policies using Human Game Play Recordings]]></title>
            <link>https://medium.com/@8B_EC/reinforcement-learning-bootstrapping-policies-using-human-game-play-recordings-ab0b02efe561?source=rss-948bd4c54478------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab0b02efe561</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[muniverse]]></category>
            <dc:creator><![CDATA[Jan Forster]]></dc:creator>
            <pubDate>Mon, 13 Nov 2017 14:01:01 GMT</pubDate>
            <atom:updated>2017-11-13T14:01:01.579Z</atom:updated>
            <content:encoded><![CDATA[<p>In my last post, I wrote about <a href="https://medium.com/@8B_EC/a-quick-look-into-%C2%B5universe-524dcf792164"><strong>µniverse</strong></a>, a great framework for training game agents using Reinforcement Learning. After training a lot of agents and playing around with µniverse, I found out that training my agent on the game <strong>StackTower </strong>is<strong> </strong>not working as good as for other games like e.g. Doodle Jump.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*ibuP6uKIYksw7BISgkgn6Q.gif" /></figure><p>The goal of <strong>StackTower </strong>is to stack as many blocks as possible on the platform. Each block fades in from east → west or north → south. Once the block reaches a <em>good </em>position, one can drop it by pressing a key. On the left you see me playing the game. I had luck with my first block, but the consecutive blocks were badly positioned.</p><p>Before we dive into the issue, the agent is facing in this game, let’s have a quick recap of Reinforcement Learning.</p><p>Reinforcement Learning consists of five basic concepts:</p><ol><li>The <strong>Agent</strong>,<strong> </strong>our actor who is looking to maximize its reward for actions in a given environment.</li><li>An <strong>Environment</strong>, which serves as a place for our actor to act in and discover, e.g. a game like Super Mario.</li><li>A <strong>State</strong>, that describes the current status of the environment.</li><li><strong>Actions</strong>, which can be performed by the agent to change the state in an environment, like jumping to get a coin in Super Mario.</li><li>The <strong>Reward</strong>, that could be earned by performing a certain action in a given state.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*TP_147a7c9d0i0xbXtM8lw.png" /><figcaption>Source: Wikimedia.com, License: CC0 1.0</figcaption></figure><p>Applied to <strong>StackTower</strong> we can see that the reward for this game is quite tricky. As humans, we intuitively know that the (long-term) reward is somehow related to our timing. The better the timing, the bigger the part of the block that remains on the platform. But this information is reflected nowhere in the game score. Whether the whole block is placed on the platform or just a part of it, the score will always be increased by one. Our agent might be able to learn this relationship, but it will take a lot of game episodes to do so.</p><p>I’ve analyzed the reward as a function of episodes for StackTower as it can be seen in the following plot. For the episodes that were played by the agent without any bootstrapping (the ones left to the black line), we can see that the reward curve stays pretty flat. Especially when assuming this growth rate for the following episodes, we could expect an average reward of 6 per episode after playing 2000 episodes (orange line). Which means that the agent is able to stack up 6 blocks on top of each other after learning from 2000 episodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*YcOSgtYgvG2SDQqRgsxUfg.png" /><figcaption>Episode Reward for Agent 1 (with and without bootstrapping)</figcaption></figure><p>On my machine this would take roughly 48 hours, so I wanted to speed up the whole process by supporting the agent. Therefore I’ve been using <a href="https://twitter.com/unixpickle">@unixpickle’s</a> <a href="https://github.com/unixpickle/demoverse">Demoverse</a> to record my gameplay. The best score I was able to achieve on my machine, was 12 stacked blocks (red line in the plot). The average stack size was around 9. So I took 10 game recordings and used them to train the agent’s policy on those. As we can see in the above chart, the agent started to rapidly increase its average reward after updating the policy (the curve right to the black line).</p><p>And this is the agent playing StackTower after training on 2000 episodes, using bootstrapping after episode 800.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*6HQp10VhEVg2AAKhNwm1rw.gif" /><figcaption>The agent playing StackTower — way better than I’m able to.</figcaption></figure><p><em>If you enjoyed this post, </em><strong><em>please click </em>👏<em> </em></strong><em>so that others can read it as well. Follow me on Twitter </em><a href="https://twitter.com/8B_EC"><em>@8B_EC</em></a><em> for the latest updates or just to say hi :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab0b02efe561" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A quick look into µniverse]]></title>
            <link>https://medium.com/@8B_EC/a-quick-look-into-%C2%B5universe-524dcf792164?source=rss-948bd4c54478------2</link>
            <guid isPermaLink="false">https://medium.com/p/524dcf792164</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[muniverse]]></category>
            <dc:creator><![CDATA[Jan Forster]]></dc:creator>
            <pubDate>Sat, 04 Nov 2017 23:23:07 GMT</pubDate>
            <atom:updated>2017-11-11T23:04:52.575Z</atom:updated>
            <content:encoded><![CDATA[<p>After working a couple of weeks with <a href="https://openai.com/">OpenAI’s</a> Gym and Universe I’m still very excited to discover and learn all possibilities to train RL agents using those frameworks.</p><p>Unfortunately, it seems that there isn’t a huge community actively using Universe. I don’t know whether that’s related to the specific topic of RL or the challenges that come up when working with Universe due to its specific architecture as mentioned e.g. by <a href="https://blog.aqnichol.com/2017/06/11/why-im-remaking-openai-universe/">Alex Nichol</a>:</p><blockquote>… the biggest problem with Universe is that VNC and Flash need to run in real time. This means that any hiccups on your training machine […] might suddenly change the frame rate at which your AI experiences its virtual environment.</blockquote><p>Therefore I was looking out for solutions to performance issues I faced when running more complex environments in Universe and came across <a href="https://github.com/unixpickle/muniverse"><strong>µniverse</strong></a>. µniverse is developed by Alex Nichol aka <a href="https://github.com/unixpickle/"><strong>unixpickle</strong></a> and providing environments to train RL agents for HTML5 games, which should improve the overall performance and complexity as RL agent training &amp; development framework.</p><h3>1 | Installation</h3><p>The installation of µniverse is pretty straightforward and documented on the µniverse Github page. <br>Unixpickle also developed a µniverse-agent which provides ready-to-use agents based on popular concepts like PPO, TRPO, A3C.<br>Available parameters can be shown by typing the following commands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*29IwHMk1NWhYiRBI_qEFnQ.png" /><figcaption>Choosing different agents for µniverse</figcaption></figure><p>Algorithm-specific parameters can be found by typing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*uNGv3_irVigtXn03igASdg.png" /><figcaption>Options for the µniverse A3C agent</figcaption></figure><h3>2 | Training an Agent</h3><p>I’ve chosen Doodle Jump as the first game to train A3C agents on, but you can choose from all games provided by µniverse. They can be found in the µniverse games folder. The naming is somehow similar to the naming conventions used in OpenAI’s Gym.<br>To start the training process, the following command can be used:</p><pre>muniverse-agent a3c -env DoodleJump-v0 -out doodlejump &gt;&gt; log_doodle.txt 2&gt;&amp;1</pre><p>The first and second parameters are pretty self-explanatory.</p><p>The out parameter provides a name where the trained policy is stored to and can be loaded from to continue training.</p><p>At the end of the command &gt;&gt; log_doodle.text 2&gt;&amp;1 is forwarding the output of the µniverse-agent to a log file we’ll use for performance analysis in a next article.</p><h3>3 | Observing the Agent</h3><p>While the training is running you might have taken a look into the muniverse-agent’s folder to search for some kind of visualization of the agent’s current state. One way to get an understanding of the agent’s performance is to take a look at the log file created in the previous step. Another way is to visualize the agent’s interactions with the environment. In our case the Doodle Jump gameplay.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*v4zUjPfqY0Ujkb2aV4NCjw.gif" /><figcaption>Agent playing Doodle Jump</figcaption></figure><p>This can simply be done by stopping the training and adding the flag -record &lt;PATH_TO_RECORDING_FOLDER&gt;. Restarting the agent will create a couple hundred images containing “screenshots” of the agent’s game play.</p><h3>4 | Next Steps</h3><p>In the next article I will show you an easy way to visualize the agent’s performance based on the logs we’ve created in this little tutorial. I’m really looking forward to see what kind of frameworks unixpickle, OpenAI and other companies will work on in the future.</p><p><em>If you enjoyed this post, </em><strong><em>please click </em>👏<em> </em></strong><em>so that others can read it as well. Follow me on Twitter </em><a href="https://twitter.com/8B_EC"><em>@8B_EC</em></a><em> for the latest updates or just to say hi :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=524dcf792164" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to setup OpenAI Universe in Windows using Docker]]></title>
            <link>https://medium.com/@8B_EC/how-to-setup-openai-universe-in-windows-using-docker-aa6409163848?source=rss-948bd4c54478------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa6409163848</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[universe]]></category>
            <category><![CDATA[starter-agent]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Jan Forster]]></dc:creator>
            <pubDate>Sun, 15 Oct 2017 17:21:26 GMT</pubDate>
            <atom:updated>2018-01-10T16:02:55.576Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*76LwzHXTcKXuU4D90ixyAQ.jpeg" /><figcaption>Source: <a href="https://www.flickr.com/photos/textfiles/27228418683/">https://www.flickr.com/photos/textfiles/27228418683/</a> licensed under the <a href="https://en.wikipedia.org/wiki/en:Creative_Commons">Creative Commons</a> <a href="https://creativecommons.org/licenses/by/2.0/deed.en">Attribution 2.0 Generic</a></figcaption></figure><p>When I first heard about <a href="https://blog.openai.com/universe/"><strong>OpenAI Universe</strong></a>, I wanted to start playing around with it as quickly as possible. Universe was released as</p><blockquote>…, a software platform for measuring and training an AI’s general intelligence across the world’s supply of games, websites and other applications.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G00jbBQIjj8ISFlH.png" /><figcaption>Source: <a href="https://blog.openai.com/universe/">https://blog.openai.com/universe/</a></figcaption></figure><p>But soon I had to realize that Windows isn’t probably the best OS to use as a starting point as it is currently not supported by <strong>OpenAI Universe</strong>. So my first naive idea was to setup up an Ubuntu VM and get the Universe Starter Agent going. Although my machine had a new i7 and 32 GB of memory the results were disappointing — even if I only ran 4 agents.</p><p>I’ve never managed to reduce the reaction time to less than 60 ms in the Pong environment.</p><p>So a guest Ubuntu wasn’t the way to go. Buying a new SSD and installing Ubuntu would have taken too much time and AWS EC2 wasn’t an alternative, so I decided to try out Docker.</p><h3>1| The setup</h3><p>So I followed the instructions on the OpenAI Universe Github page:<br>To get started, open a Docker Quickstart Terminal as Admin and clone the universe repo:</p><pre>git clone <a href="https://github.com/openai/universe.git">https://github.com/openai/universe.git</a><br>cd universe</pre><p>Build a docker image, tag it as ‘universe’:</p><pre>docker build -t universe .</pre><p>If the error “<em>Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23)”.</em> pops up, you may have to run the following command to circumvent an issue due to an older Docker Toolbox Installation on Windows 7:</p><pre>export DOCKER_API_VERSION=1.23</pre><p>The next step is to start a container from universe image. The -p switches are used to redirect the ports of Tensorboard and the VNC outputs to access them from your Windows machine later on.</p><pre>docker run --privileged --rm -it -p 12345:12345 -p 5900:5900 -e DOCKER_NET_HOST=172.17.0.1 universe /bin/bash</pre><p>Because of the -it switch and the /bin/bash command you should find yourself in a pseudo tty pointing to the directory /usr/locasl/universe</p><p>A prerequisite for installing the <strong>universe-starter-agent</strong> is Miniconda.</p><p>Download the latest installer using wget in your container:</p><pre>wget <a href="https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh">https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh</a></pre><p>Start the installation typing:</p><pre>bash ./Miniconda3-latest-Linux-x86_64.sh</pre><p>and confirm the prompts.</p><p>Once Miniconda is installed, follow the installation instructions given on <a href="https://github.com/openai/universe-starter-agent"><em>https://github.com/openai/universe-starter-agent</em></a></p><pre>conda create --name universe-starter-agent python=3.5<br>source /root/minconda3/bin/activate universe-starter-agent</pre><pre>apt-get install -y tmux htop cmake golang libjpeg-dev</pre><pre>pip install “gym[atari]”<br>pip install universe<br>pip install six<br>pip install tensorflow<br>conda install -y -c <a href="https://conda.binstar.org/menpo">https://conda.binstar.org/menpo</a> opencv3<br>conda install -y numpy<br>conda install -y scipy</pre><p>Now you have everything installed to start using the <strong>universe-starter-agent</strong>. It’s a good time to commit the changes to your universe docker image.</p><p>Run docker ps to get the container id, then run</p><pre>docker commit &lt;CONTAINER_ID&gt; universe</pre><h3>2| The test run</h3><p>Make sure you’re in the universe-starter-agent folder and in the correct conda environment by running the following command in your container bash:</p><pre>cd universe-starter-agent &amp;&amp; source /root/miniconda3/bin/activate universe-starter-agent</pre><p>Start Pacman to see if everything is working correctly by running:</p><pre><br>python train.py — num-workers 2 — env-id gym-core.MsPacman-v0 — log-dir /tmp/pacman</pre><p>Now you should see a similar output on your screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tSeY21Hrb7M0nQHwiqp1gA.png" /><figcaption>Pacman session with 2 workers starting in container</figcaption></figure><p>You can now issue the command tmux a to browse through the different sessions that were started to make sure the workers are running correctly. When starting the workers for the first time, they’ll pull a container with all the necessary tools needed, which might take a couple of minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*j4aLHqt3mSiIJeuKi-qPSw.png" /><figcaption>tmux screen</figcaption></figure><p>In tmux you can change between the different processes (workers, tensorboard, orchestrator and htop) by entering <strong>CTRL+b [0–4]</strong>. Return to bash by pressing <strong>CTRL+b d</strong></p><h3><strong>3| Watching your agents play</strong></h3><p>If the workers are running correctly and your agents start training, you can view the details in Tensorboard from your Windows machine’s browser. The only thing you need is the IP your Docker is running with. You can see the default machine’s IP when starting a new Docker Quickstart Terminal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*tZdjxWs8MxNO9ATUrgDfDQ.png" /><figcaption>Docker Quickstart Terminal startup screen for default machine on Windows 7</figcaption></figure><p>Just enter &lt;YOUR_DOCKER_IP&gt;:12345 and Tensorboard should opened.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKrK7OlKeNuia6sIlOj4sQ.png" /></figure><p>To view the agents playing, use the VNC viewer of your choice and connect it to one of the VNC ports. You can get ports by running docker ps in your Docker Terminal. This returns a list of your primary container and the child containers with your workers. Each worker routes its VNC port 5900 to a free port e.g. 5901 as shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uUScUg-a0sjLF8Rg7pxa4Q.png" /><figcaption>Worker VNC ports (yellow)</figcaption></figure><p>When connecting to a VNC output, enter <strong>openai </strong>as password and you should see the agent playing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*hJZTXGZI4cUfIr8YmvLBYg.png" /><figcaption>Worker 1 playing Pacman</figcaption></figure><p>Entering tmux kill-session will stop your workers. If you want to reuse your models in a next training session, think of mounting your Windows filesystem to docker to transfer the <em>log-dir</em> contents to your local machine.</p><p>I hope my little how-to supported you to get OpenAI universe running on your Windows machine.</p><p><em>If you enjoyed this post, </em><strong><em>please click </em>👏<em> </em></strong><em>so that others can read it as well. Follow me on Twitter </em><a href="https://twitter.com/8B_EC"><em>@8B_EC</em></a><em> for the latest updates or just to say hi :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa6409163848" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>