<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alessandro Solbiati on Medium]]></title>
        <description><![CDATA[Stories by Alessandro Solbiati on Medium]]></description>
        <link>https://medium.com/@alessandrosolbiati?source=rss-50fab72dd23d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*s47vCtvDxJUTrLMFO3zy4w.jpeg</url>
            <title>Stories by Alessandro Solbiati on Medium</title>
            <link>https://medium.com/@alessandrosolbiati?source=rss-50fab72dd23d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:56:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alessandrosolbiati/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Policy Gradients in Reinforcement Learning, Homework 2 — CS294]]></title>
            <link>https://medium.com/cs294-112-berkley-deep-reinforcement-learning/policy-gradients-in-reinforcement-learning-homework-2-cs294-75a77756c355?source=rss-50fab72dd23d------2</link>
            <guid isPermaLink="false">https://medium.com/p/75a77756c355</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Alessandro Solbiati]]></dc:creator>
            <pubDate>Sat, 20 Jul 2019 10:06:13 GMT</pubDate>
            <atom:updated>2019-07-27T21:36:43.744Z</atom:updated>
            <content:encoded><![CDATA[<h3>Policy Gradients in Reinforcement Learning, Homework 2 — CS294</h3><p>This are our notes to the <a href="http://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw2.pdf">Homework 2</a> for the course <a href="http://rail.eecs.berkeley.edu/deeprlcourse/">CS294–112 Berkley — Deep Reinforcement Learning</a>. This document is organised as follows:</p><ol><li><strong>Lecture Review:</strong> review of the theoretical concepts, here are some english language comments to the formula</li><li><strong>Problem 1:</strong> we try to prove that a state-dependent value function is an unbiased baseline for the policy gradient.</li></ol><h3>1. Lecture Review</h3><h4>1.1 Formula (1) : objective function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEcZWACSvLE_7o94y8SAIA.png" /></figure><p>(1) This is the objective function, tau is a trajectory. A trajectory is a sequence of (observation, action).<br>`tau ~ pisubtetha` means that the trajectories are distributed as the probability distribution `pi` that comes out of the current policy `p`. If you compare to throwing dices, tau is the outcome of how you throw the dice and pi_theta is the distribution that tells you how the dice behave.<br>`pisubtheta(tau)` is the probability that the given trajectory tau will happen.<br>the reward of a trajectory `r(tau)` is just the sum of all the rewards of the states on the trajectory<br>More formally, `π(θ)` is a probability distribution over the action space, conditioned on the state. <br>In the agent-environment loop, the agent samples an action at from πθ(·|st) and the environment responds with a reward r(st,at).</p><h4>1.2 Formula (2) : gradient definition</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kq-M4Kvk0ZsklN7GjjqFfw.png" /></figure><p>We simply take the gradient of the objective function and we apply gradient descent. This is just some multivariable calculus math.<br>How do we compute it? Using a batch of trajectories</p><h4>1.3 Formula (4): gradient computation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ErUf6lM6tZeLR_rUvGe4g.png" /></figure><p>The computation of the gradient in batches is just the gradient in the classical supervised learning problem, just weighted by the reward.<br>Not super clear yet how to compute since we have `pisubtheta(tau)` where tau is a trajectory.</p><h4>1.4 Formula (5): gradient computation with exploded trajectories</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_zr6gF0mQpKXvY6o-XFV_A.png" /></figure><p><em>This is the key section of the whole policy gradient lecture.</em></p><p>We are going through batches of trajectories tau . So here we explode the trajectories so we can understand better what pisubtheta means.<br>This formula is important for the computation so we might as well spend a bit more time on it. Let’s examine the single parts.</p><p>1. `∇θ log πθ( action| state)`, here we are taking the gradient of the logarithm of `pi`. This is because `pisubtheta` is not a number but is a probability distribution. In common English, the meaning of this expressions is:<strong> how should I change my parameters </strong><strong>theta in order to increase the probability that the next time I will be in </strong><strong>state I will take </strong><strong>action .</strong></p><p>2.`r(state, action), here we sum all the reward and we get the score of the given trajectory.</p><p><strong><em>Key Insight on Policy Gradient</em></strong><em>: </em>The policy gradient term and the reward term are multiplied because <em>I am not sure the next time I am in this state I actually want to take this action. </em>If I really want to do it is told to me by the r(state, action). If the reward is negative, I will actually decrease the probability of taking this action again. If the reward is positive, I will increase the probability of taking this action again.</p><p>What we are doing here is summing the gradients of all the steps of the trajectory and then multiplying by the `total reward of the trajectory.`<br>What does this mean? Well in supervised learning (when you compute the gradient of the log loss), that gradient tells you where to move towards your truth label, that’s because the log loss is computed for the truth labels. Here the policy `pi` is not the truth label, so our `pi` gradient <strong>does not tell us a priori that the gradient direction is the improvement direction</strong>. To know if it is the correct direction we need to multiply by the total trajectory reward: that’s how we know the improvement direction.</p><p>This is one of the key concepts of reinforcement learning. One problem we have though is that there is too much variance over the reward term: it could change a lot from trajectory to trajectory, and for this reason, we would lose signal on how good our gradient.</p><h4>1.5 Formula (6) : reward-to-go</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9i3ZRzf4eFRLhjDqFhIBFw.png" /></figure><p>Looks like now we are moving the sum of rewards inside the sum of gradients. The idea here is that I am <strong>multiplying step by step the current gradient by the reward to go</strong>.<br>This reduces the variance, you can imagine you are assigning your ‘truth labels’ in a less generic and more accurate way compared to before.</p><h4>1.6 Formula (7) and (8): discount factor</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LKSf9IqtO2A-7mxBVjFCbA.png" /></figure><p>We multiply the reward at every step `t1` by a discount factor `gamma ** (t1 — t)`. This reduces the importance of future rewards since they are less influenced by the gradient at the current steps. This explains (8) whereas is not clear how this is useful in applying the discount factor to the whole trajectory like in (7).</p><h4>1.7 Formula (9): baseline subtraction</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDoNaOVdHmsWVlhYMIwMxQ.png" /></figure><p>We can subtract a baseline constant with respect to `tau` (so it is constant in all the trajectory `tau`, but is not necessarily constant in time `t` of the simulation). <br>This also reduced variance as explained in lecture. Why? This is a interesting point. Here you are multiplying by the gradient, but what happens if you reward is always positive? The model will think that what he is doing is always correct, in the sense that step by step the gradient will be multiplied by a positive return, thus it will indicate that the gradient direction is the improvement direction.<br>In this case we are actually giving the model a <strong>wrong signal</strong>, and a good work around can be subtracting a baseline. The <strong>baseline is chosen to be the average of the rewards for the trajectory</strong> so that your single rewards will be correctly calibrated in being positive or negative (since we subtract the average).</p><h4>1.8 Formula (10): value function as a baseline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AOZkwzk2T84M_DrO-QQHJQ.png" /></figure><p>In the homework we use a <strong>state-dependent baseline</strong>: it’s a value function as it is a function of a state (it expresses the value of a particular state). It is defined as the the sum of future rewards starting from a particular state, given the current policy.</p><p><strong>Why we use the value function?</strong> The value function is the expectation of how good is the state you are currently in, i.e. how much reward you will get from there. So intuitively it makes sense: you are in a state, and you want to subtract (as a baseline) what you already know about that state (how good it is).</p><p>An interesting point here is that actually, we don’t know if this baseline is unbiased with respect to the gradient: in the lecture was proven that a baseline that is trajectory independent is unbiased. The homework will go through this later.</p><h4>1.9 Formula (11): final policy gradient expression</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gVfUsjrI0BZVVMut8kkP5w.png" /></figure><p>This just puts everything together: we get the baseline of (6) of the sums of gradients multiplied step by step by the reward to go. There is the (8) discount factor and the (9) baseline subtraction.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75a77756c355" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cs294-112-berkley-deep-reinforcement-learning/policy-gradients-in-reinforcement-learning-homework-2-cs294-75a77756c355">Policy Gradients in Reinforcement Learning, Homework 2 — CS294</a> was originally published in <a href="https://medium.com/cs294-112-berkley-deep-reinforcement-learning">CS294–112 Berkley — Deep Reinforcement Learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CS294 Deep Reinforcement Learning]]></title>
            <link>https://medium.com/series/cs294-deep-reinforcement-learning-13416ca0b4b7?source=rss-50fab72dd23d------2</link>
            <guid isPermaLink="false">https://medium.com/p/13416ca0b4b7</guid>
            <dc:creator><![CDATA[Alessandro Solbiati]]></dc:creator>
            <pubDate>Sat, 20 Jul 2019 09:59:25 GMT</pubDate>
            <atom:updated>2019-07-20T09:59:25.589Z</atom:updated>
            <content:encoded><![CDATA[<p>Homework walkthrough of the course <a href="http://rail.eecs.berkeley.edu/deeprlcourse/">CS294–112 Berkley — Deep Reinforcement Learning</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13416ca0b4b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Starter Guide to VnPy]]></title>
            <link>https://medium.com/@alessandrosolbiati/starter-guide-to-vnpy-911a81539302?source=rss-50fab72dd23d------2</link>
            <guid isPermaLink="false">https://medium.com/p/911a81539302</guid>
            <category><![CDATA[vnpy]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <dc:creator><![CDATA[Alessandro Solbiati]]></dc:creator>
            <pubDate>Sun, 03 Mar 2019 18:46:32 GMT</pubDate>
            <atom:updated>2019-03-03T18:46:32.211Z</atom:updated>
            <content:encoded><![CDATA[<h3>VnPy Starter Guide</h3><p><strong>Vn.py —a Python-based algo-trading development framework</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*IEVKple86rX2fyoRUhpKSg.png" /></figure><h3>What is VnPy?</h3><p>VnPy is an algorithmic trading development framework, with a bunch of awesome features:</p><ul><li>completely <strong>open-source</strong></li><li>written in <strong>Python</strong></li><li>with an <strong>active community</strong> of more than 5000 people</li></ul><p>Nonetheless, you probably never heard about it before. This is because of the <em>fourth</em> awesome feature:</p><ul><li>the community is mostly <strong>Chinese-based</strong> and Chinese-speaking</li></ul><p>The VnPy project was started in 2015 in Shanghai by the first generation of options traders in China. It has grown ever since to a terrific open-source project that legitimately created the biggest and <strong>most active algo-traders community in China. </strong>This series of guides will be an effort to make this vast amount of knowledge <strong>available to the English speaking community,</strong> in a structured and organized way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F04q3EgXLCXTYAUCNYm7rA.png" /></figure><h3>Why should I choose VnPy as a development framework for my algo-trading strategies?</h3><p>Some of the reasons why I chose VnPy over other frameworks</p><ul><li><strong>Community:</strong> VnPy is the <em>third biggest algo-trading framework project on GitHub</em> at the current moment (by stars count). The community is huge and is rare to see such active communities around: questions are being answered on the various chat almost<em> </em>immediately<em> </em>at any time.</li><li><strong>Open Source: </strong>works <em>right out of the box</em> as a prototyping tool without studying the various details. At the same time, I can dive into the source code to ensure security and flexibility, perfect if you want to integrate it into your existing systems or develop additional features by yourself.</li><li><strong>Python: </strong>being Python based means I can easily integrate all my favourite <em>libraries</em> like Pandas, Scipy, Numpy, Matplotlib, Sci-kit, Tensorflow and Keras. I am mainly interested in Machine-Learning applications to algo-trading and this is the only framework that gives me full flexibility and adaptability (most of the others like Quantopian or Meta-Trader don’t allow the use of external machine-learning libraries).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*-603H6U32pt876bY1pHByQ.png" /></figure><h3>What are the current resources available online regarding VnPy?</h3><ul><li><a href="https://github.com/vnpy/vnpy">GitHub repository</a>, with the complete codebase</li><li><a href="https://github.com/vnpy/vnpy/wiki">GitHub wiki</a>, pretty exhaustive</li><li>the <a href="http://www.vnpy.org/">official website</a>, (in Chinese)</li><li><a href="/service/qq:262656087">QQ Group</a>, a Chinese telegram-like group with incredibly active members</li><li><a href="http://www.vnpie.com/forum.php">vnpie.com</a>, old-style bulletin forum in Chinese about VnPy</li><li><a href="https://zhuanlan.zhihu.com/vn-py">ZhiHu page</a> of VnPy project founder, a Chinese Quora-like platform with questions and answers about VnPy</li></ul><h3>Ok, I want to give it a try, how do I get started?</h3><p>There are plenty of Starter Guides online on VnPy, but unfortunately all in Mandarin. If lately, you didn’t brush up on your Chinese, I will walk you through the <strong>first steps* of setting up your first algo-trading strategy</strong> on VnPy using <strong>Interactive Broker</strong>.</p><p>What you will need:</p><ul><li>30 minutes of your time and internet connection for setting up VnPy and start trading</li><li>Ubuntu 16.04 (following instructions on how to set it up on a Virtual Machine), 15 minutes for the download and set-up + 1.65GB</li><li>Interactive Broker account or demo-account</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j4CRG6z7K29xwgOFzQiHUQ.png" /></figure><h3><strong>Step 0— Getting the right OS</strong></h3><p>The most practical way to get up to speed with VnPy is to run an Ubuntu 16.04 LTS virtual machine on your local computer. This will <strong>save you a lot of troubles during the installation</strong>, especially when building the Interactive Broker API (tested only for Windows from the IB guys). Ubuntu 16.04 is the reference version for the VnPy installation and ideally everything will run smoothly without unexpected errors. You can choose your favourite Virtual Machine application. I always use <strong>V</strong><a href="https://www.virtualbox.org/wiki/Downloads"><strong>irtual Box</strong></a><strong>.</strong></p><p>Installing Ubuntu is pretty straight-forward:</p><ul><li>download the .iso of Ubuntu 16.04 image at this link <a href="http://releases.ubuntu.com/16.04/">http://releases.ubuntu.com/16.04/</a>, is around 1.65GB so it might take a while to download based on your connection</li><li>run a new virtual box on Virtual Box (Machine-&gt;New) and just follow the instructions. You shouldn&#39;t have any problem, if you get stuck you can follow this post from <a href="https://askubuntu.com/questions/142549/how-to-install-ubuntu-on-virtualbox">askubuntu</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*sikQNlno3bekzr-jhXediQ.jpeg" /><figcaption>When the installation is completed it should look something like this.</figcaption></figure><h3>Step 01— Getting Python</h3><p>Now we need to install pip, Python and the project dependencies. In practice we just need to <strong>copy paste a lot of apt-get commands into the terminal</strong>, you can open the terminal with [Control]+[Option]+T (on Mac Keyboard).</p><p>Let’s first set a source for <strong>pip</strong>, that will be used by Python to install packages.</p><pre>mkdir ~/.pip/<br>sudo apt-get install nano<br>nano ~/.pip/pip.conf</pre><p>You can use this mirror for pip, copy-paste the following lines inside<em> pip.conf</em></p><pre>[global]<br>Index-url = https://pypi.tuna.tsinghua.edu.cn/simple</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*Zf5OgFbb8tX8YJ3UfUpZxg.png" /></figure><p>Now that we have pip, let’s get Python. We will install Anaconda (that already contains a lot of interesting stuff). Download Anaconda2 and install it, execute the following 3 lines of commands sequentially in the terminal.</p><pre>Wget <a href="https://repo.continuum.io/archive/Anaconda2-4.0.0-Linux-x86_64.sh">https://repo.continuum.io/archive/Anaconda2-4.0.0-Linux-x86_64.sh</a></pre><pre>Chmod +x Anaconda2-4.0.0-Linux-x86_64.sh</pre><pre>./Anaconda2-4.0.0-Linux-x86_64.sh</pre><p>Then start the installation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*9JdjWIMs1s3Wbtr9" /></figure><p>Press Enter at the prompt, and then enter Yes when prompted to update the PATH variable in the .bashrc file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/0*tMGEgPn0y4n25jF2" /></figure><p>After the installation is complete, execute the following command to make the bash configuration file take effect immediately.</p><pre>Source  ~ /.bashrc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*BRa76jCs3wWRsu9lpijumQ.png" /><figcaption>Once installed Anaconda, you should type ‘python’ on the terminal and see this. If you had any problem you can refer to Anaconda <a href="https://docs.anaconda.com/anaconda/install/">official docs</a>.</figcaption></figure><h3>Step 02 — Getting Dependencies</h3><p>Run the following commands to install the project dependencies.</p><pre>pip install future</pre><pre>sudo apt-get install mongodb<br>sudo apt-get install libboost-all-dev<br>sudo apt-get install cmake<br>sudo apt-get install git <br>sudo apt-get install libsnappy-dev<br>sudo apt-get install python-snappy</pre><pre>sudo apt-get install build-essential<br>sudo apt-get install python-dev</pre><p>As you can see, the DB system used by VnPy is mongo DB. You can also download a desktop GUI application to interact with the database. In the community the usually recommend <a href="https://robomongo.org/">robomongo</a>. You can postpone this later.</p><h3>Step 03 — Get VnPy and run the VnTrader</h3><p>Just clone the git repository and run the installer</p><pre>git clone <a href="https://github.com/vnpy/vnpy">https://github.com/vnpy/vnpy</a><br>cd vnpy</pre><p>Now you will be inside the VnPy master folder. You can run the installer with</p><pre>bash install.sh</pre><p>You will be asked about what interfaces (brokers) you want to build, we will stick only to our Interactive Broker one for now (since you won’t be probably trading with Chinese brokers).</p><pre>Do you need &#39;CTP&#39; interface? No<br>Do you need &#39;LTS&#39; interface? No<br>Do you need &#39;XTP&#39; interface? No<br>Do you need &#39;IB&#39; interface? Yes</pre><p>When is successfully built, you can enter into examples and run VnTrader.</p><pre>cd vnpy/examples/VnTrader<br>python run.py</pre><p>You should get by default some ‘module not found’ errors since you didn’t build all the broker interfaces, you just need to comment a couple of imports inside run.py as mentioned by this question on StackOverflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iwi_ZhsXBZFadSqNw81DhA.png" /><figcaption>Congrats! You are running VnPy on Ubuntu!</figcaption></figure><p>You would probably want to change the language to English, you just need to go into ‘examples/VT_setting.json’ and change the language (check<a href="https://stackoverflow.com/questions/53703851/vnpy-does-vntrader-has-an-english-version"> this question</a> on Stack Over Flow)</p><h3>Step 04 — Connect Interactive Brokers</h3><p>So now we are running VnPy successfully, we need to connect Interactive Brokers to actually place our orders. We will quickly create a demo-account and create a connection. If you are not familiar with Interactive Brokers APIs there is a lot of interesting <a href="http://interactivebrokers.github.io/tws-api/">contents on the web</a>, you won’t really need to know much since the interaction is pretty much incorporated inside VnPy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vFFhlY9HijZ2jeV9qr564w.png" /><figcaption>TWS from Interactive Broker</figcaption></figure><p>We need to download a demo version of the Trader Work Station (TWS). It will act as a “local server” listening to our calls from VnTrader (“local client”). There is nothing complicated, just follow the instructions at the <a href="https://www.interactivebrokers.com/en/index.php?f=19903&amp;invd=T">Interactive Broker TWS download page</a></p><p>Once is downloaded, you need to <a href="http://interactivebrokers.github.io/tws-api/initial_setup.html">allow API connections</a>, and take notes of your account-id (in light-blue in the image above). Insert your account-id inside <em>vnpy/examples/VnTrader/IB_connect.json</em></p><pre>{<br> &quot;host&quot;: &quot;localhost&quot;,<br> &quot;port&quot;: 7497,<br> &quot;clientId&quot;: 888,<br> &quot;accountCode&quot;: &quot;DU229585&quot;<br>}</pre><p>Now we are ready to connect VnTrader to IB TWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YhRy9vGnvyTJ6OE1Hufayw.png" /><figcaption>VnTrader connected to IB, left-bottom ‘Logs’, right-bottom ‘Positions’</figcaption></figure><ul><li>Open TWS and make sure it allows incoming APIs calls</li><li>Run VnTrader (python run.py inside examples/VnTrader)</li><li>Inside VnTrader: System -&gt; Connect IB, and Connect DB</li><li>In the down-left box, select ‘Log’ to check that the connection was successful.</li></ul><p>How can you test your connection? Just place an order inside TWS (in my example I bought 100.0 MSFT stocks), and the Position box in VnTrader should update accordingly! <strong>Magic</strong>! (check the picture above)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=911a81539302" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>