<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nicola Russo on Medium]]></title>
        <description><![CDATA[Stories by Nicola Russo on Medium]]></description>
        <link>https://medium.com/@nicrusso7?source=rss-8caf8613a8fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZhsGLUHxkUKNNBmb.jpg</url>
            <title>Stories by Nicola Russo on Medium</title>
            <link>https://medium.com/@nicrusso7?source=rss-8caf8613a8fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:07:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nicrusso7/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Training a Spot Inspired Quadruped Robot Using Reinforcement Learning]]></title>
            <link>https://medium.com/swlh/training-a-spot-inspired-quadruped-robot-using-reinforcement-learning-678b9e5df164?source=rss-8caf8613a8fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/678b9e5df164</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[robotics]]></category>
            <dc:creator><![CDATA[Nicola Russo]]></dc:creator>
            <pubDate>Tue, 15 Sep 2020 20:15:30 GMT</pubDate>
            <atom:updated>2020-09-16T22:55:59.717Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s been a while since I’ve started exploring Reinforcement Learning and OpenAI Gym , inspired by the amazing Boston Dynamics Spot. I’ve spent last year studying the foundations of Machine Learning and how it is applied to robotics discovering a very interesting world.</p><p>In order to experiment what I’ve learned, I’ve created an open-source project called <a href="https://github.com/nicrusso7/rex-gym"><strong>rex-gym</strong></a><strong>. </strong>The aim is to let an open-source quadruped robot learns domestic and generic tasks in the simulations and then successfully transfer the knowledge (Control Policies) on the real robot without any other manual tuning.</p><p>Rex is a 12 joints robot with 3 motors (Shoulder, Leg and Foot) for each leg. The robot base model is imported in pyBullet using a <a href="https://github.com/nicrusso7/rex-gym/blob/master/rex_gym/util/pybullet_data/assets/urdf/rex.urdf">URDF file</a> and the servo motors are modelled in the motor class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*kONyzEJp5-vgSqQUfn4Dlg.png" /><figcaption>Rex simulation</figcaption></figure><p>There is also an enhanced version that comes with a 6DOF robotic arm mounted on the top of the rack</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*bB053pDRm6JG4RdhoeFv6A.png" /><figcaption>Rex with a 6DOF robotic arm</figcaption></figure><h3>Learning approach</h3><p>This library uses the Proximal Policy Optimization (PPO) algorithm with a hybrid policy defined as:</p><p>a(t, o) = a(t) + π(o)</p><p>It can be varied continuously from fully user-specified to entirely learned from scratch. If we want to use a user-specified policy, we can set both the lower and the upper bounds of π(o) to be zero. If we want a policy that is learned from scratch, we can set a(t) = 0 and give the feedback component π(o) a wide output range.</p><p>By varying the open loop signal and the output bound of the feedback component, we can decide how much user control is applied to the system.</p><p>A twofold approach is used to implement the Rex Gym Environments: Bezier controller and Open Loop.</p><p>The Bezier controller implements a fully user-specified policy. The controller uses the Inverse Kinematics model (see model/kinematics.py) to generate the gait.</p><p>The Open Loop mode consists, in some cases, in let the system lean from scratch (setting the open loop component a(t) = 0) while others just providing a simple trajectory reference (e.g. a(t) = sin(t)).</p><p>The purpose is to compare the learned policies and scores using those two different approach.</p><h3>Gaits</h3><p>My first focus was on the basic gaits: walk, gallop, turn and stand up.</p><p>I’ve created a gym environment for each of them (tuning the action space and the reward function ) and I’ve trained the robot using both the Bezier controller and Open Loop one. The output are different but efficient gaits:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Ma6QMUfi8cSg5Va00uFGRQ.gif" /><figcaption>Bezier: Walk</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Hb1C5zfgCX3ujw5x625Z7g.gif" /><figcaption>Bezier: Walk backwards</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*9a5UlByHuZj-9IJy1HHYNg.gif" /><figcaption>OpenLoop: Walk</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*LT2RgZbC68FTnXe2p9z1Jw.gif" /><figcaption>Bezier: Gallop</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*E7MjuG1Ckkd6dpHRtyg2JQ.gif" /><figcaption>OpenLoop: Gallop</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Bfy_zKNIsSWTefx84BlaIQ.gif" /><figcaption>Bezier: Turn</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*v37GCQEJQxLsxpeNv6bDEA.gif" /><figcaption>OpenLoop: Turn</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*fMevbQ1K_yA8HEv1mDvfyQ.gif" /><figcaption>OpenLoop: Stand up</figcaption></figure><h3>Terrains</h3><p>In order to test the policies’ robustness, I’ve integrated different uneven terrains other than the classic plane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*pcE2MeGGkea3HSG4qOCBnQ.gif" /><figcaption>Random heightfield</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Dt8pFn8nMNSybRkE_shxEg.gif" /><figcaption>Hills</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*KMWpYfJHBXUjVgok4-ftqQ.gif" /><figcaption>Mounts</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*7gYClTvgbriBCSIRMz18Kg.gif" /><figcaption>Maze</figcaption></figure><p>The next step will be training the robot to interact with the environments, grasping objects and opening doors.</p><p>Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=678b9e5df164" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/training-a-spot-inspired-quadruped-robot-using-reinforcement-learning-678b9e5df164">Training a Spot Inspired Quadruped Robot Using Reinforcement Learning</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>