<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jacob Gursky on Medium]]></title>
        <description><![CDATA[Stories by Jacob Gursky on Medium]]></description>
        <link>https://medium.com/@gurskyjacob?source=rss-13efc0f9bee7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*1MGnj-uMTg--Vwnvre7R8w.jpeg</url>
            <title>Stories by Jacob Gursky on Medium</title>
            <link>https://medium.com/@gurskyjacob?source=rss-13efc0f9bee7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:27:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gurskyjacob/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Easily Improve Any GAN with Metropolis-Hastings]]></title>
            <link>https://medium.com/data-science/easily-improve-any-gan-with-metropolis-hastings-bf7a021785f1?source=rss-13efc0f9bee7------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf7a021785f1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Jacob Gursky]]></dc:creator>
            <pubDate>Thu, 21 May 2020 02:35:36 GMT</pubDate>
            <atom:updated>2020-05-21T13:38:03.229Z</atom:updated>
            <content:encoded><![CDATA[<h4>Create higher fidelity samples with no additional training in TensorFlow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*zYcuwNDi-xhFhYdC.jpg" /><figcaption>How can we improve our GAN with as little work as possible? (Image from <a href="https://pixabay.com/photos/fox-sleeping-resting-relaxing-red-1284512/">Pixabay</a>)</figcaption></figure><p>Ever had trouble getting believable outputs from your GAN? In this article we are going to review a recent improvement to GANs that boosts output quality with an easy-to-follow Tensorflow 2 implementation to get your own Metropolis-Hastings GAN off the ground!</p><h3>About the Paper</h3><p>In late 2018, Uber Engineering released a paper entitled<strong> </strong>“Metropolis- Hastings Generative Adversarial Networks”<strong> </strong>(Turner et al.), in which the authors introduce a simple trick that improves the output of GANs with <strong>no additional train time</strong> of the GAN. If you are not very familiar with GANs, I suggest reading this <a href="https://towardsdatascience.com/understanding-generative-adversarial-networks-gans-cd6e4651a29">article</a> before proceeding to get a better understand of the models we are dealing with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*IvzvggHPsTgDbWiYZiKmwA.png" /><figcaption>A visual from “Metropolis-Hastings Generative Adversarial Networks” (2018, Turner et al.) showing how MH can improve the outputs of Progressive GANs</figcaption></figure><p>The basic premise of Uber’s work is that the discriminator, which is normally thrown out after training, contains valuable information in determining which samples resemble the true data distribution. The authors propose a novel sampling scheme in which samples are drawn from the latent space of the generator using Metropolis-Hastings with the discriminator choosing which generated samples to accept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*3CvQqMIL7N72JsNveZdpUg.png" /><figcaption>Figure 1: The Metropolis-Hastings GAN pseudocode from the Uber paper</figcaption></figure><p>What’s ingenious about this method is that it works with <strong>any GAN architecture or loss function</strong>, as MH-GAN is really just a post-training sampling methodology. Before we show how to apply this sampling method, let’s first train a GAN to use it on!</p><h3>Training our GAN in TensorFlow 2</h3><p>To start on our journey of training a MH-GAN, we are first going to a train a Least-Squares GAN due to its stability and ease of implementation. I included a link at the bottom of the article if you want to learn more about LS-GANs. For the data, we are going to be working with the Fashion MNIST dataset included in TensorFlow so these results are easy to replicate at home!</p><p>Please note that some of the training code below is adapted from the official TensorFlow DC-GAN tutorial located <a href="https://www.tensorflow.org/tutorials/generative/dcgan">here</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/667e5f0de9cd67d552033976388a568b/href">https://medium.com/media/667e5f0de9cd67d552033976388a568b/href</a></iframe><p>Note that we are also using the tensorflow-addons package to implement some more advanced functionality such as the Ranger optimizer and Mish activation layers to smooth the training process.</p><h4>Defining Our Generator and Discriminator</h4><p>The next step in training our networks is to define our generator and discriminator objects to feed into our LS-GAN class:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6276b568d0f1b6c943e618b86b5f9af9/href">https://medium.com/media/6276b568d0f1b6c943e618b86b5f9af9/href</a></iframe><p>We have a base class to train a Least-Squares GAN as well as a generator and discriminator to feed into it, now its time to dive into implementing the MH-GAN!</p><h3>The Metropolis-Hastings Algorithm</h3><p>The first thing we need to understand when implementing a MH-GAN is the Metropolis-Hastings algorithm. MH is a Markov-Chain Monte-Carlo method for sampling from a distribution that is difficult to directly sample from. In this case, we are trying to sample from the true data distribution that is implied by the discriminator.</p><p>We won’t dive too deep into the theory behind Metropolis-Hastings, but I have included a link at the bottom of this article if you want to learn more.</p><p>Once we have a trained generator and discriminator, we use the following steps to draw samples that follow the true data distribution (feel free to refer to the pseudocode above):</p><ol><li>Seed the chain with a real sample of data (the authors do this to avoid the lengthy burn-in period that can be entailed with Metropolis-Hastings)</li><li>For K number of iterations, we draw a random sample from the generator and calculate the score from the discriminator</li><li>At each iteration, if the following expression is greater than a random number in the range [0,1] then the new sample is accepted and compared to the next iteration:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*aR-cSGEqwbesVcc9Che19g.png" /><figcaption>Figure 2: Condition for accepting a new sample in an MH-GAN (from the Uber MH-GAN paper)</figcaption></figure><p>Essentially what this rule is doing is comparing the ratio of the proposed sample’s discriminator score to the current sample’s score to determine if we should “accept” the new sample. With enough iterations our independent chain will approximate the true data distribution implied by our discriminator’s output!</p><p>If we fail to accept any samples after a certain number of iterations, the authors recommend restarting the chain from a synthetic sample to guarantee convergence, albeit at a slower pace.</p><p><strong>This is not the end of the story however</strong>, as there is an outstanding issue with the output of our discriminator, which we will address in the next section.</p><h3>Calibrating Our Discriminator</h3><p>One of the assumptions made in applying Metropolis-Hastings is that the probabilities of our discriminator are well-calibrated. This means that the discriminator output must be interpretable as a confidence level. Unfortunately, the raw outputs of a neural network classifier are almost never well-calibrated, meaning we have to apply an additional step to calibrate the outputs. <strong>This is true even if we use a sigmoid or softmax activation on the final layer!</strong></p><p>The Scikit-Learn documentation has a great section on calibration <a href="https://scikit-learn.org/stable/modules/calibration.html">here</a>, but what the authors of the paper recommend is using a technique called isotonic regression to convert the network outputs to calibrated probabilities as a post-processing step. Unfortunately, this needs to be done on a held-out set, meaning we have to divide our Fashion-MNIST into a training and validation set (fortunately Tensorflow does this for us!)</p><p>Below is a calibration curve (a diagnostic tool that illustrates the calibration of a model, <strong>the 45 degree line represents perfect calibration</strong>) on held-out data showing how our isotonic predictions satisfy the condition of Metropolis-Hastings. Our raw outputs, on the other hand, are clearly poorly-calibrated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*nCx2D5FfhAVaaNNAvMLp5Q.png" /><figcaption>Figure 3: Calibration curve showing improvement after applying isotonic regression to discriminator outputs</figcaption></figure><p>Once we have our discriminator calibrated, we can then use the calibrated probabilities in determining the acceptance probabilities of new samples in the independent MCMC chain we outlined above!</p><h3>Coding our MH-GAN</h3><p>Now its finally time to put the calibration and Metropolis-Hastings sampling into practice with a subclass of the LS-GAN we coded above. Note that we could easily apply this subclass to any other sort of GAN that follows the API we outlined above, or even adapt it to existing GAN packages.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d8272d0e815d290d2f6dbbfb9280244/href">https://medium.com/media/2d8272d0e815d290d2f6dbbfb9280244/href</a></iframe><p>Note that there are a couple of things going on here, first we need to implement a method to train our calibrator as well as a calibrated scoring method.</p><p>For training the calibrator, we simply take a held-out set and generate the same number of samples from the generator. Next, we train the IsotonicRegression class from Scikit-Learn to translate the discriminator scores into true probabilities.</p><p>Finally, we have our generate_mh_samples() method, which generates a specified number of accepted samples given a true example to seed the MCMC chain. Can you see a common theme here? The calibrated scoring and MH sample generation methods are simply the MH-GAN counterparts to the score_samples() and generate_samples() methods from the LS-GAN!</p><p>Time to wet our feet with this new class using the Fashion-MNIST data!</p><h3>Comparing the Results</h3><p>To start using our shiny new MH-GAN class, the first thing we have to do is load our Fashion-MNIST data and separate it into our training and validation sets, as well as normalizing the pixels. Fortunately, tf.keras makes this super easy!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6e928d95946f5044f29829c9d38b1dc/href">https://medium.com/media/d6e928d95946f5044f29829c9d38b1dc/href</a></iframe><p>Now that we have our training and test sets, all we have to do is initialize our MHGAN object with the generator and discriminator we declared above, as well as the dimensionality of our noise for the generator and the optimizer learning rate. We are going to then train for 50 epochs (this was rather arbitrary, usually this needs to be more thought-out for GANs), and finally calibrate our discriminator. Our object-oriented approach allows us to do this in 3 lines of code!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3623ff24afa3b8f3d97eaf5d830596b1/href">https://medium.com/media/3623ff24afa3b8f3d97eaf5d830596b1/href</a></iframe><p>After the training is completed, we can draw samples from our trained MH-GAN by using either our normal naive GAN sampling, or the Metropolis-Hastings algorithm from the Uber paper.</p><p>Let’s take a look at 15 images generated using LS-GAN, MH-GAN, and some true images to compare:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*r_5p0TAJrAXccpzmzUz60g.png" /><figcaption>Figure 4: Comparison of LS-GAN and MH-GAN outputs with true samples from the Fashion-MNIST dataset</figcaption></figure><p><strong>Wow! </strong>We can definitely see that the MH-GAN nearly entirely avoids the issue of clearly fake samples! This finding is in line with that of the Uber team, who reported on CelebA dataset their method drastically reduces the number of low-fidelity images. We can see instances of samples from the LS-GAN that just look like ill-defined blobs, whereas this occurs much less frequently in our MH-GAN.</p><p>The team also argues that it addresses the mode-collapse problem, but in my trials of their method I didn’t really see this occurring, but MNIST and Fashion-MNIST aren’t always the best indicators of algorithm performance and this wasn’t an issue to begin with.</p><h3>Conclusions</h3><p>If you are interested in applying the quick improvement to GANs we demonstrated in this article, <strong>I would highly recommend</strong> reading the original MH-GAN paper which is linked below! Furthermore, we have shown that TensorFlow 2 makes it extremely easy to apply the technique with very little additional code!</p><p>There are quite a few other intriguing papers regarding discriminator-based GAN sampling, such as <a href="https://arxiv.org/abs/1810.06758">“Discriminator Rejection Sampling”</a> by Azadi et al. and <a href="https://arxiv.org/abs/2003.06060">“Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling”</a> by Che et al., showing the promise of this field. No doubt it will be interesting to see where this goes over the next few years.</p><p>If you have any questions or comments about this article, please feel free to post them below!</p><h3>References and Resources</h3><p>Link to the original MH-GAN paper by Uber:</p><p><a href="https://arxiv.org/abs/1811.11357">Metropolis-Hastings Generative Adversarial Networks</a></p><p>Original website article by Uber Engineering on MH-GANS:</p><p><a href="https://eng.uber.com/mh-gan/">How to Get a Better GAN (Almost) for Free: Introducing the Metropolis-Hastings GAN</a></p><p>A helpful article on Least-Squares GAN</p><p><a href="https://wiseodd.github.io/techblog/2017/03/02/least-squares-gan/">Least Squares GAN</a></p><p>A quick tutorial to Metropolis-Hastings sampling:</p><p><a href="https://towardsdatascience.com/from-scratch-bayesian-inference-markov-chain-monte-carlo-and-metropolis-hastings-in-python-ef21a29e25a">From Scratch: Bayesian Inference, Markov Chain Monte Carlo and Metropolis Hastings, in python</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf7a021785f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/easily-improve-any-gan-with-metropolis-hastings-bf7a021785f1">Easily Improve Any GAN with Metropolis-Hastings</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[100% Faster Reinforcement Learning Environments with Cygym]]></title>
            <link>https://medium.com/data-science/100-faster-reinforcement-learning-environments-with-cygym-20a3a970ebd1?source=rss-13efc0f9bee7------2</link>
            <guid isPermaLink="false">https://medium.com/p/20a3a970ebd1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[openai-gym]]></category>
            <dc:creator><![CDATA[Jacob Gursky]]></dc:creator>
            <pubDate>Wed, 08 Apr 2020 01:16:17 GMT</pubDate>
            <atom:updated>2020-04-08T16:20:35.725Z</atom:updated>
            <content:encoded><![CDATA[<h4>Faster drop-in replacements for Gym environments using cygym</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*pNvFijAo7SkVzZJR" /><figcaption>Can we speed up reinforcement learning with faster environments? <a href="https://unsplash.com/photos/dDp-6pXrYik">Image Source</a></figcaption></figure><p>Anyone who at least dabbles in reinforcement learning will likely tell you they have used OpenAI’s Gym package (linked below), and for good reason! It is an easy-to-use, extensible, and well-supported package that reduces much of the overhead in setting up a RL project.</p><p>However, when conducting a large-scale project with minimal resources, its essential that every drop of performance be squeezed out of code that is run many times. While working on an RL project, I took a look through the source code of the classic control environments that are often used for benchmarking RL agents to see if we could get some cheap speed gains.</p><p>I noticed that all of the gym code is written in pure Python, so the production-code part of my brain immediately went: “What about cythonizing these environments for a speed boost?”</p><h4>Quick Intro to Cython</h4><p>Cython is a superset of Python that enables compilation of interpreted Python code into extremely fast C extensions. What’s nice about this is that we don’t actually have to write any C code, only some minor modifications to our Python script! Depending on the use-case, sometimes you can see 1000x+ speed increases over Python code. The .step() method of gym environments seemed like a good candidate for this, as it is called once for every frame of a game, which can occur millions of times in training an RL agent.</p><h3>Introducing cygym</h3><p>I recently released a public repository containing a Python package called <strong>cygym</strong>, which contains cythonized versions of the classic control environments from the original gym package. All credit for the original environment code goes to the authors of those scripts, but I did introduce some minor optimizations, as well as leveraging some Cython magic to get up to a 100% speed boost over the original implementations. Hopefully, this number will only go up as we expand our code base and get more contributors.</p><h3>Installation and Importing</h3><p>Installation and setup of cygym are quite easy, simply clone the public package repository and install using pip!</p><pre>git clone https://github.com/gursky1/cygym<br>cd cygym<br>pip install -e . </pre><p>Pip should handle all of the setup needed for the package, but in the event that you have issues with installation that way, you can manually build the package yourself and install in an alternative way using setuptools:</p><pre>python setup.py build_ext --inplace bdist_wheel<br>pip install dist/(wheel file here).whl</pre><p>If you are still running into issues, please feel free to open an issue or contact me directly!</p><h3>Quick Start</h3><p>Cygym is nearly a complete drop-in for OpenAI gym, all you have to do is import the environment classes and get started!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/daba892d53b57cab73466a89842e19d0/href">https://medium.com/media/daba892d53b57cab73466a89842e19d0/href</a></iframe><p>Cygym should also be compatible with any and all OpenAI gym wrappers. One feature we are working on right now is getting the cythonized classes compatible with the OpenAI gym registry, but for Cython reasons this is not super straightforward.</p><h3>Some Initial Speed Comparisons</h3><p>One of the first speed tests we can perform is comparing how long each of the two most common gym methods, .reset() and .step() , take to run in both gym and cygym for our classic control problems.</p><p>.reset() is called when we initialize an environment, while .step() is called to apply an action and get the next frame. Most of the run time in interacting with an environment is spent in .step(), so we will place more weight on that comparison:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*YBD6kJ9ny4LSgBoM6_4grw.png" /><figcaption>Absolute and relative speed comparisons between gym and cygym for initializing environments</figcaption></figure><p>Unfortunately, it seems that cythonizing only brings a moderate speed boost to initializing environments using .reset(), and sometimes even slows it down! But fear not, this method is only called once per episode, so these results are secondary to our next comparison:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*dqzDy9EO4TKE32JVyqURhg.png" /><figcaption>Absolute and relative speed comparisons between gym and cygym for iterating environments</figcaption></figure><p>Wow! We can see up to a six times speed boost in iterating our environments using our cythonized variants! Note that most of our environment interaction time is here, meaning our overall model training time should be less dependent on the environment!</p><p>Testing two common methods with random actions is all well and good, but how does cygym do in actually training agents? Let’s take a look…</p><h3>RL Agent Training with Stable Baselines</h3><p>Stable baselines is an excellent fork of the original OpenAI baselines package that contains implementations of numerous RL algorithms such as Deep-Q, TRPO, SAC, PPO, and many more. I won’t dive too deep into this package, but we will be utilizing the vectorized environments available in Stable-Baseslines with our gym and cygym environments to train actual RL agents, and see how long it takes to train our models for 100k frames:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7668549be904b3b30e84583d9a9d92ae/href">https://medium.com/media/7668549be904b3b30e84583d9a9d92ae/href</a></iframe><p>Now we are going to run a similar comparison as the previous sections, looking at all of our classic control environments to see what kind of speed up we can expect using cygym:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*_7eBSBTZVCBUsF1NxOkRzQ.png" /><figcaption>Absolute and relative speed comparisons between gym and cygym for training RL agents</figcaption></figure><p>Again, wow! It looks like cygym was always faster than gym, with up to a 50% reduction in train time! This can be a massive difference when trying to prototype new model architectures or doing hyperparameter searches.</p><h3>Future Releases</h3><p>In the future we plan to look at further compatibility with OpenAI gym, such as using gym.make() with our cythonized environments, and possibly fast rendering (which currently isn’t supported).</p><p>We also plan to take a look at cythonizing more complex tasks such as those in the box2d and algorithms modules of the original gym package.</p><h3>Conclusions</h3><p>We have shown that simple cythonization with cygym can lead to huge decreases in RL agent train time, meaning less time waiting and more time analyzing results! We are actively looking for contributors and maintainers for cygym, so if you have experience in Cython package development and optimization, please get in touch!</p><h3>Resources</h3><p><strong>Cygym: Fast gym-compatible classic control RL environments</strong></p><p><a href="https://github.com/gursky1/cygym">gursky1/cygym</a></p><p><strong>OpenAI Gym: A versatile package for reinforcement learning environments</strong></p><p><a href="https://github.com/openai/gym">GitHub - openai/gym: A toolkit for developing and comparing reinforcement learning algorithms.</a></p><p><strong>Cython: A compiled superset of the Python language</strong></p><p><a href="https://cython.org/">Cython: C-Extensions for Python</a></p><p><strong>Stable-Baselines: A fork of OpenAI baselines</strong></p><p><a href="https://github.com/hill-a/stable-baselines">hill-a/stable-baselines</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20a3a970ebd1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/100-faster-reinforcement-learning-environments-with-cygym-20a3a970ebd1">100% Faster Reinforcement Learning Environments with Cygym</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gradient-Free Reinforcement Learning: Neuroevolution using Numpy!]]></title>
            <link>https://medium.com/data-science/gradient-free-reinforcement-learning-neuroevolution-using-numpy-7377d6f6c3ea?source=rss-13efc0f9bee7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7377d6f6c3ea</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Jacob Gursky]]></dc:creator>
            <pubDate>Tue, 07 Apr 2020 23:53:29 GMT</pubDate>
            <atom:updated>2020-04-08T02:56:09.050Z</atom:updated>
            <content:encoded><![CDATA[<h4>Can we solve simple RL environments with no backward passes?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*NkQrtUHTsTg9ykn2.png" /><figcaption>Can we train networks with only forward passes? <a href="https://pixabay.com/vectors/evolution-walking-charles-darwin-297234/">Image Source</a></figcaption></figure><h3>Introduction</h3><p>What if I told you that you can train neural networks without ever calculating a gradient, and only using the forward pass? Such is the magic of <strong>neuroevolution! </strong>Also, I am going to show that all this can easily be done using only Numpy! Studying statistics you learn so much about gradient-based methods, but a while back I read a really interesting article by the folks at Uber AI, who showed that a simple genetic algorithm was competitive with the most sophisticated gradient-based RL methods at solving Atari games in terms of wall-clock time. I linked the source below, I highly recommend you give it a read if you are interested in reinforcement learning.</p><h3>What is Neuroevolution?</h3><p>First off, for those of you that don’t already know, <strong>neuroevolution</strong> describes the application of evolutionary and/or genetic algorithms to training either the structure and/or weights of neural networks as a gradient-free alternative! We are going to use an extremely simple case of neuroevolution here, only using a fixed topology network and focusing on optimizing only weights and biases. The neuroevolutionary process can be defined by four fundamental steps that are repeated until convergence is reached, starting with a pool of randomly generated networks.</p><p>1. Evaluate fitness of the population<br>2. Select the most fit individual to reproduce<br>3. Repopulate using copies of the most fit network<br>4. Introduce normally distributed mutations to the network weights</p><p>Wow, this seems pretty simple! Let’s break down some of the terminology a bit:</p><p>- <strong>Fitness</strong>: This simply describes how well the network performed at a certain task and allows us to determine which networks to breed. Note that because evolutionary algorithms are a form of non-convex optimization, and <strong>therefore can be used with any loss function</strong>, regardless of its differentiability (or lack thereof)</p><p>- <strong>Mutation</strong>: This one is probably the easiest! In order for our child networks to improve, we have to introduce random changes to the network weights, often drawn from a uniform or normal distribution. There can be many different forms of mutation: shift mutations (which multiply the parameters by a random number), swap mutations (which replace the parameter with a random number), sign mutations (which change the sign of a parameter), etc. We are only going to be using simple additive mutations, but there is a lot of room for creativity here!</p><h3>Advantages of Neuroevolution</h3><p>We should also consider the theoretical advantages of neuroevolutionary modeling. First off, we only need to use the forward pass of the network as we only need to calculate the loss in order to determine which networks to breed. The implications of this are obvious, the backwards pass is usually the most expensive! Second, evolutionary algorithms are guaranteed to find the global minimum of a loss surface given enough iterations, whereas convex gradient-based methods can get stuck in local minima. Lastly, more sophisticated forms of neuroevolution allow us to not only optimize the weights of a network, but also the structure itself!</p><h3>Why Not Always Use Neuroevolution Then?</h3><p>Well, this is a complicated question but it really boils down to exact gradient descent methods being more effective when enough gradient information is available. This means that the more convex a loss surface is, the more you will want to use analytic methods like SGD rather than genetic algorithms. As a consequence, it is very rare that you will use genetic algorithms in a supervised context, as usually there is enough gradient information available that traditional gradient descent methods will work quite well. However, if you are working in a RL context, or in a case with irregular loss surfaces or low convexity (like a sequential GAN), then neuroevolution provides a viable alternative! In fact, much research has come out lately finding that parameter-for-parameter neuroevolutionary models can do better in these settings.</p><h3>Now let’s jump in!</h3><h4>Loading Libraries</h4><p>As laid out in the introduction, we are going to try and use <strong>only</strong> numpy for this project, only defining the helper functions that we need! Yes, I know, gym is also being loaded, but only for the environment ;)</p><pre>import numpy as np<br>import gym</pre><h4>About the Data</h4><p>We are going to use the classic CartPole environment from gym to test our networks. The goal is to see how long the network can keep the pole upright by shifting left and right. As a RL task, neuroevolutionary methods should be a good fit! Our network will take in 4 observations as input, and will output either left or right as an action.</p><h4>Helper Functions</h4><p>We first are going to define a few of the helper functions to set up our networks. First off is the relu activation function, which we will use as the activation function for our hidden layers, and the softmax function for the output of the network to get probabilistic estimates of the network output! Lastly, we need to define a function that generates one-hot encodings of our response vector for when we need to calculate categorical cross-entropy.</p><pre>def relu(x):<br>    return np.where(x&gt;0,x,0)</pre><pre>def softmax(x):<br>    x = np.exp(x — np.max(x))<br>    x[x==0] = 1e-15<br>    return np.array(x / x.sum())</pre><h4>Defining Our Network Class</h4><p>Now comes the fun stuff! First, we are going to define a class for our individual networks within the population. We need to define an initialization method that randomly assigns weights and biases and takes the network structure as input, a prediction method so we can get probabilities given an input, and finally an evaluation method that returns the categorical cross-entropy of the network given an input and response! Again, we are only going to use functions we define or functions from numpy. Note that the initialization method can also take another network as an input, this is how we will perform mutations between generations!</p><pre># Lets define a new neural network class that can interact with gym<br>class NeuralNet():<br>    <br>    def __init__(self, n_units=None, copy_network=None, var=0.02, episodes=50, max_episode_length=200):</pre><pre>        # Testing if we need to copy a network<br>        if copy_network is None:<br>            # Saving attributes<br>            self.n_units = n_units<br>            # Initializing empty lists to hold matrices<br>            weights = []<br>            biases = []</pre><pre>            # Populating the lists<br>            for i in range(len(n_units)-1):<br>                weights.append(np.random.normal(loc=0,scale=1,size=(n_units[i],n_units[i+1])))<br>                biases.append(np.zeros(n_units[i+1]))<br>            # Creating dictionary of parameters<br>            self.params = {&#39;weights&#39;:weights,&#39;biases&#39;:biases}</pre><pre>        else:<br>            # Copying over elements<br>            self.n_units = copy_network.n_units<br>            self.params = {&#39;weights&#39;:np.copy(copy_network.params[&#39;weights&#39;]),<br>                          &#39;biases&#39;:np.copy(copy_network.params[&#39;biases&#39;])}<br>            # Mutating the weights<br>            self.params[&#39;weights&#39;] = [x+np.random.normal(loc=0,scale=var,size=x.shape) for x in self.params[&#39;weights&#39;]]<br>            self.params[&#39;biases&#39;] = [x+np.random.normal(loc=0,scale=var,size=x.shape) for x in self.params[&#39;biases&#39;]]<br>            </pre><pre>    def act(self, X):<br>        # Grabbing weights and biases<br>        weights = self.params[&#39;weights&#39;]<br>        biases = self.params[&#39;biases&#39;]<br>        # First propgating inputs<br>        a = relu((X@weights[0])+biases[0])<br>        # Now propogating through every other layer<br>        for i in range(1,len(weights)):<br>            a = relu((a@weights[i])+biases[i])<br>        # Getting probabilities by using the softmax function<br>        probs = softmax(a)<br>        return np.argmax(probs)<br>        <br>    # Defining the evaluation method<br>    def evaluate(self, episodes, max_episode_length, render_env, record):<br>        # Creating empty list for rewards<br>        rewards = []</pre><pre>        # First we need to set up our gym environment<br>        env=gym.make(&#39;CartPole-v0&#39;)</pre><pre>        # Recording video if we need to <br>        if record is True:<br>            env = gym.wrappers.Monitor(env, &quot;recording&quot;)</pre><pre>        # Increasing max steps<br>        env._max_episode_steps=1e20<br>        for i_episode in range(episodes):<br>            observation = env.reset()<br>            for t in range(max_episode_length):<br>                if render_env is True:<br>                    env.render()<br>                observation, _, done, _ = env.step(self.act(np.array(observation)))<br>                if done:<br>                    rewards.append(t)<br>                    break<br>        # Closing our enviroment<br>        env.close()<br>        # Getting our final reward<br>        if len(rewards) == 0:<br>            return 0<br>        else:<br>            return np.array(rewards).mean()</pre><h4>Defining Our Genetic Algorithm Class</h4><p>Lastly we need to define a class that manages our population, performing the four key steps in neuroevolution! We need three methods here. First, an initialization method that creates a pool of random networks and sets attributes. Next, we need a fit method that, given an input, repeatedly performs the steps outlined above: first evaluating networks, then selecting the most fit, creating child networks, and finally mutating the children! Lastly, we need a prediction method so that we can use the best network trained by the class. Let’s get down to testing!</p><pre># Defining our class that handles populations of networks<br>class GeneticNetworks():<br>    <br>    # Defining our initialization method<br>    def __init__(self, architecture=(4,16,2),population_size=50, generations=500,render_env=True, record=False,<br>                 mutation_variance=0.02,verbose=False,print_every=1,episodes=10,max_episode_length=200):<br>        # Creating our list of networks<br>        self.networks = [NeuralNet(architecture) for _ in range(population_size)]<br>        self.population_size = population_size<br>        self.generations = generations<br>        self.mutation_variance = mutation_variance<br>        self.verbose = verbose<br>        self.print_every = print_every<br>        self.fitness = []<br>        self.episodes = episodes<br>        self.max_episode_length = max_episode_length<br>        self.render_env = render_env<br>        self.record = record<br>        <br>    # Defining our fiting method<br>    def fit(self):<br>        # Iterating over all generations<br>        for i in range(self.generations):<br>            # Doing our evaluations<br>            rewards = np.array([x.evaluate(self.episodes, self.max_episode_length, self.render_env, self.record) for x in self.networks])<br>            # Tracking best score per generation<br>            self.fitness.append(np.max(rewards))<br>            # Selecting the best network<br>            best_network = np.argmax(rewards)<br>            # Creating our child networks<br>            new_networks = [NeuralNet(copy_network=self.networks[best_network], var=self.mutation_variance, max_episode_length=self.max_episode_length) for _ in range(self.population_size-1)]<br>            # Setting our new networks<br>            self.networks = [self.networks[best_network]]+new_networks<br>            # Printing output if necessary<br>            if self.verbose is True and (i%self.print_every==0 or i==0):<br>                print(&#39;Generation:&#39;,i+1,&#39;| Highest Reward:&#39;,rewards.max().round(1),&#39;| Average Reward:&#39;,rewards.mean().round(1))<br>        <br>        # Returning the best network<br>        self.best_network = self.networks[best_network]</pre><h4>Testing Our Algorithm!</h4><p>As stated above, we are going to test our networks on the CartPole problem, using only 1 hidden layer with 16 nodes, and two output nodes denoting either a left or right movement. We also need to average over many episodes, so we don’t accidentally pick a bad network for the next generation! I picked many of these parameters after a little trial-and-error, so your mileage may vary! Also, we will only introduce mutations with a variance of 0.05, so as to not break the functionality of the networks.</p><pre># Lets train a population of networks<br>from time import time<br>start_time = time()<br>genetic_pop = GeneticNetworks(architecture=(4,16,2),<br>                                population_size=64, <br>                                generations=5,<br>                                episodes=15, <br>                                mutation_variance=0.1,<br>                                max_episode_length=10000,<br>                                render_env=False,<br>                                verbose=True)<br>genetic_pop.fit()<br>print(&#39;Finished in&#39;,round(time()-start_time,3),&#39;seconds&#39;)</pre><pre>Generation: 1 | Highest Reward: 309.5 | Average Reward: 29.2<br>Generation: 2 | Highest Reward: 360.9 | Average Reward: 133.6<br>Generation: 3 | Highest Reward: 648.2 | Average Reward: 148.0<br>Generation: 4 | Highest Reward: 616.6 | Average Reward: 149.9<br>Generation: 5 | Highest Reward: 2060.1 | Average Reward: 368.3<br>Finished in 35.569 seconds</pre><h4>Initial Random Network</h4><p>First, let’s look at how a randomly initialized network performs on the task. Obviously, there is no strategy here and the pole falls over almost immediately. Please ignore the cursor in the gif below, recording in Gym doesn’t play nice with Windows!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/0*-uQ_lzcmbOZXwKWl" /><figcaption>Figure 1: A random agent trying to solve CartPole</figcaption></figure><pre>random_network = NeuralNet(n_units=(4,16,2))<br>random_network.evaluate(episodes=1, max_episode_length=int(1e10), render_env=True, record=False)</pre><h4>5 Generations Later…</h4><p>After only 5 generations, we can see that our network has almost completely mastered the art of the CartPole! And it only took around thirty seconds of train-time! Note that with further training, the network learns to keep it completely upright nearly 100% of the time, but for now we are just interested in speed, and 5 generations is rather short! We should consider this a good example of the power of neuroevolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/0*cQlSNEmTvZPJ8xfM" /><figcaption>Figure 2: A trained agent solving the CartPole task</figcaption></figure><pre># Lets observe our best network<br>genetic_pop.best_network.evaluate(episodes=3, max_episode_length=int(1e10), render_env=True, record=False)</pre><h3>What&#39;s Next?</h3><p>Obviously, there are quite a few things we could add in the future to further examine the effectiveness of neuroevolution. Firstly, it would be interesting to study the effects of different mutation operators such as cross-over.</p><p>It would also be a smart idea to shift over to a modern deep-learning platform like TensorFlow or PyTorch. Note that genetic algorithms are highly parallelizable, as all we would have to do is run each network on a single device with one forward pass. No need to mirror weights or complicated distribution strategies! Therefore we have a nearly linear decrease in run-time with each additional unit of processing.</p><p>Lastly, we should explore neuroevolution on different reinforcement learning tasks, or even other situations where gradients can be difficult to evaluate, such as in Generative Adversarial Networks or long-sequence LSTM networks.</p><h3>Further Reading</h3><p>If you are interested in neuroevolution and its applications, Uber has a fantastic page on several papers showing the modern advantages of neuroevolution in reinforcement learning:</p><p><a href="https://eng.uber.com/tag/deep-neuroevolution/">Uber Engineering Blog</a></p><p>The source code for this project can be found at my public GitHub repository:</p><p><a href="https://github.com/gursky1/Numpy-Neuroevolution">gursky1/Numpy-Neuroevolution</a></p><p>If you have any questions about this project or just want to discuss deep learning, feel free to email me at <a href="/service/mailto:gurskyjacob@gmail.com">gurskyjacob@gmail.com</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7377d6f6c3ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gradient-free-reinforcement-learning-neuroevolution-using-numpy-7377d6f6c3ea">Gradient-Free Reinforcement Learning: Neuroevolution using Numpy!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boosting Showdown: Scikit-Learn vs XGBoost vs LightGBM vs CatBoost in Sentiment Classification]]></title>
            <link>https://medium.com/data-science/boosting-showdown-scikit-learn-vs-xgboost-vs-lightgbm-vs-catboost-in-sentiment-classification-f7c7f46fd956?source=rss-13efc0f9bee7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7c7f46fd956</guid>
            <category><![CDATA[gradient-boosting]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jacob Gursky]]></dc:creator>
            <pubDate>Mon, 30 Mar 2020 04:40:49 GMT</pubDate>
            <atom:updated>2020-04-28T21:48:09.686Z</atom:updated>
            <content:encoded><![CDATA[<h4>Which gradient boosting library will reign supreme in this head-to-head competition?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dc86g87niJNRaPlQ8AoaqQ.jpeg" /><figcaption><a href="https://pixabay.com/photos/arm-wrestling-indian-wrestling-567950/">Image Source</a></figcaption></figure><p><strong>Gradient boosting</strong> is one of the most popular machine learning techniques in recent years, dominating many Kaggle competitions with heterogeneous tabular data. Similar to random forest (if you are not familiar with this ensembling algorithm I suggest you read up on it), gradient boosting works by ensembling many decision trees in order to perform regression or classification. However, unlike random forest, gradient boosting grows trees sequentially, iteratively growing trees based on the residuals of the previous tree. Doing so allows gradient boosting to focus on particularly tricky observations and yields an extraordinarily powerful ensemble of trees.</p><p>This is a gross oversimplification that does not speak to the variety of boosting implementations that exist, but that is not the purpose of this article. If you are interested in reading more about the specifics of how boosting works, I have attached some useful resources at the bottom of the article.</p><p>The gradient boosting space has become somewhat crowded in recent years with competing algorithms such as XGBoost, LightGBM, and CatBoost vying for users. The three just mentioned are among the most common to see in Kaggle competitions, due to their speed, power, and GPU-compatibility. There have been a few articles already comparing these boosting variants on heterogeneous data, such as:</p><ul><li>CatBoost vs Light GBM vs XGBoost:</li></ul><p><a href="https://towardsdatascience.com/catboost-vs-light-gbm-vs-xgboost-5f93620723db">CatBoost vs. Light GBM vs. XGBoost</a></p><ul><li>XGBOOST vs LightGBM: Which algorithm wins the race !!!</li></ul><p><a href="https://towardsdatascience.com/lightgbm-vs-xgboost-which-algorithm-win-the-race-1ff7dd4917d">LightGBM vs XGBOOST: Which algorithm win the race !!!</a></p><p>There has not been, however, any analysis comparing these competitors in other settings such as text classification. There is a good reason for this, as gradient boosting isn’t considered to be the most appropriate algorithm for these sort of problems, but it will still be interesting to see how this comparison plays out!</p><h3>About the Data</h3><p>We are going to analyze the Large Movie Review Dataset available from Stanford (linked below), which contains 50,000 movie reviews associated with a “positive” or “negative” label. The data is already separated into training and testing sets of 25,000 reviews each. Our labels are also perfectly balanced, with 25,000 positive and 25,000 negative reviews total. We should acknowledge how clean this is, as in text classification we rarely have this level of class balance!</p><h3>Preparing Our Data</h3><p>As our emphasis here is comparing gradient boosting methods, we are going to perform simple preprocessing on our text data, just coercing it into a data frame, performing TFIDF-vectorization, a common method for converting a variable-length text into a bag-of-words representation representing transformed word frequencies in said text. If you are not familiar with this technique, again I have placed a link at the bottom of the article, as this is a must-know in the field of NLP. We are going to restrict the number of words in our preprocessed data frames to use the 2,000 top words in terms of TFIDF frequencies.</p><p>We are also going to split our training set into a training and validation set so that we can perform early-stopping, a regularization technique, on some of our boosting candidates that don’t have functionality to perform this split internally.</p><pre># Importing packages and data<br>from time import time<br>import numpy as np<br>import pandas as pd<br>from sklearn.model_selection import train_test_split<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>from sklearn.svm import LinearSVC<br>from sklearn.tree import DecisionTreeClassifier<br>from sklearn.experimental import enable_hist_gradient_boosting<br>from sklearn.ensemble import (RandomForestClassifier,<br>                              AdaBoostClassifier,<br>                              GradientBoostingClassifier,<br>                              HistGradientBoostingClassifier)<br>from xgboost import XGBClassifier<br>from lightgbm import LGBMClassifier<br>from catboost import CatBoostClassifier<br>import matplotlib.pyplot as plt</pre><pre>df_train = pd.read_csv(&#39;train.csv&#39;).sample(frac=1.0)<br>df_test = pd.read_csv(&#39;test.csv&#39;).sample(frac=1.0)</pre><pre># Preprocessing our data<br>tfidf = TfidfVectorizer(max_features=2000)<br>X_train = tfidf.fit_transform(df_train[&#39;text&#39;]).toarray()<br>X_test = tfidf.transform(df_test[&#39;text&#39;]).toarray()<br>y_train, y_test = df_train[&#39;negative&#39;], df_test[&#39;negative&#39;]<br>X_train_sub, X_valid, y_train_sub, y_valid = train_test_split(X_train, y_train, test_size=0.1)</pre><pre># Setting up our results dataframe<br>df_results = pd.DataFrame(columns=[&#39;accuracy&#39;, &#39;run_time&#39;])</pre><h3>Setting Up Our Candidate Models</h3><p>We are going to analyze a few different baseline models as well as our boosting methods, each containing a brief summary, before we dive into the full comparison on our sentiment classification task. Note that all of the accuracies and run times will be recorded in the final visuals. We are only instantiating our classifiers in this section.</p><h4>Decision Tree</h4><p>As our first baseline we are going to consider the most simple case: a single decision tree classifier. We are going to use the Scikit-Learn implementation of the famous CART algorithm to grow a classifier level-wise to classify the sentiment. Note there is a lot of tuning that we could do here, such as regularization, tree pruning, and feature sampling, but for our use-case we are simply going to specify a max depth for speed (otherwise, with 2,000 feature and unlimited depth, our tree will grow excessively deep, overfit drastically, and take forever to train).</p><pre>dt = DecisionTreeClassifier(max_depth=12, random_state=1234)</pre><h4>Random Forest</h4><p>Next, we are going to consider one of the most influential ensemble algorithms of the past few decades: random forest. Random forest works by creating a “forest” of decision trees, each trained on a bagged (bootstrap aggregated) sample of the training data. I’ll stick with this surface level overview of RF for now; if you are interested, there are resources at the bottom of the page. Random forest is remarkably good at preventing overfitting and tends to work well right out of the box. We will use 500 trees in our forest with unlimited depth as a stronger baseline for performance than our single decision tree.</p><p>Note, however, that all of our boosting algorithms (other than AdaBoost) from this point on will have a parameter similar to max_features which denotes how to subsample features at each tree or split (differs based on implementation), which we will set at 0.06, which means that we will be subsampling 6% of our features at each of these intervals. This is done to increase speed, reduce overfitting, and ensure a fair comparison between our candidate models. We will also restrict distributed computation to 6 cores for a fair comparison as well (even though some of these algorithms may scale differently with additional cores).</p><pre>rf = RandomForestClassifier(n_estimators=500,<br>                            max_features=0.06,<br>                            n_jobs=6,<br>                            random_state=1234)</pre><h4>Adaboost</h4><p>Adaboost (short for <strong>Ada</strong>ptive <strong>Boost</strong>ing) was one of the first major boosting techniques introduced way back in 1997. Adaboost is a special case of gradient boosting, minimizing the exponential loss using a series of any desired weak learners (decision stumps are often the choice here). Despite much success in earlier years, this algorithm is not seen quite as often in competitive environments, as it tends to be outperformed by more modern boosting techniques.</p><p>If you are interested in learning the differences between Adaboost and gradient boosting, I have posted a link at the bottom of this article.</p><pre>base_estim = DecisionTreeClassifier(max_depth=1, max_features=0.06)                            ab = AdaBoostClassifier(base_estimator=base_estim,<br>                        n_estimators=500,<br>                        learning_rate=0.5,<br>                        random_state=1234)</pre><h4>Gradient Boosted Decision Trees (Scikit-Learn)</h4><p>Now we will dive into the first real gradient boosting method: gradient boosted trees. Gradient boosting is a generalization of the aforementioned Adaboost algorithm, where any differentiable loss function can be used. Whereas Adaboost tries to use observation weights to inform training, gradient boosting tries to follow a gradient. This class is much more feature-rich in Scikit-Learn; we can specify subsetting the training data for regularization and select a feature subsetting percentage similar to random forest. Note that while n_estimators is set to 2000, we do not expect to get anywhere near there, and the early-stopping will stop growing new trees when our internal validation error has not improved for 15 iterations. We will follow this same early stopping procedure for all of the boosting methods to follow.</p><p>Note that it is possible to do early stopping with most iterative algorithms, but we are not using it with AdaBoost as it is not included in the Scikit-Learn implementation.</p><pre>gbm = GradientBoostingClassifier(n_estimators=2000,<br>                                 subsample=0.67,<br>                                 max_features=0.06,<br>                                 validation_fraction=0.1,<br>                                 n_iter_no_change=15,<br>                                 verbose=0,<br>                                 random_state=1234)</pre><h4>XGBoost</h4><p>Most serious ML practitioners are already familiar, or at least have heard of XGBoost (short for E<strong>X</strong>treme <strong>G</strong>radient <strong>Boost</strong>ing). XGBoost is a gradient boosting library that came to fame after winning the Kaggle Higgs Machine Learning Challenge in 2014. This library became popular for several reasons:</p><ol><li>It implemented very effective regularization hyperparameters on the underlying trees.</li><li>It was built to be highly scalable, utilizing distributed computing to process extremely large data sets.</li><li>It is available in many different languages (such as Python, R, C, C++, Ruby, Julia, etc.).</li></ol><p>XGBoost still does quite well in modern competitions, and the XGBoost community has done a great job of maintaining the packages and adding new features. However, it is facing some competition from other boosting libraries such as LightGBM and CatBoost. One disadvantage of the XGBoost library is that it doesn’t have any special treatment of categorical features like its competitors. This is unfortunate, as boosting libraries tend to excel in tabular data, which often includes discrete features. However, in our comparison this will not matter, as our TFIDF representation contains only continuous value features.</p><p>Note also that are using similar parameters here to the GradientBoostingClassifier, but also specifying tree_method=&#39;hist’. This will bin the continuous features using histogram binning, reducing the amount of splits that need to be analyzed, a technique first introduced by LightGBM.</p><pre>xgb = XGBClassifier(n_estimators=2000,<br>                    tree_method=&#39;hist&#39;,<br>                    subsample=0.67,<br>                    colsample_level=0.06,<br>                    verbose=0,<br>                    n_jobs=6,<br>                    random_state=1234)</pre><h4>LightGBM</h4><p>In my opinion, LightGBM is the reigning king of gradient boosting libraries as of March 2020. Introduced by Microsoft in 2017, LightGBM is a ridiculously fast toolkit designed for modeling extremely large data sets of high dimensionality, often being many times faster than XGBoost (though this gap was reduced when XGBoost added its own binning functionality). LightGBM attains this speed through:</p><ol><li>Being high distributed, similar to XGBoost, by design.</li><li>Using leaf-wise rather than level-wise splits when growing trees.</li><li>Using histogram binning of continuous features (the first boosting algorithm to do so) to reduce the number of splits.</li></ol><p>LightGBM also includes special support for categorical features by using a special optimal split algorithm.</p><pre>lgbm = LGBMClassifier(n_estimators=2000,<br>                      feature_fraction=0.06,<br>                      bagging_fraction=0.67,<br>                      bagging_freq=1,<br>                      verbose=0,<br>                      n_jobs=6,<br>                      random_state=1234)</pre><h4>CatBoost</h4><p>The newest of the popular gradient boosting libraries, CatBoost (<strong>Cat</strong>egorical <strong>Boost</strong>ing) was developed by the Russian tech company Yandex in mid-2017, following closely on the heels of LightGBM. Unfortunately, I have yet to see CatBoost consistently outperform its competitors (though with many categorical features it does tend to come out on top), nor match the speed of LightGBM, but that could definitely change with future updates. However, CatBoost was meant for cases such as categorical and text data, so please take the results of this article with a grain of salt when deciding which methods to try for your use-case.</p><p>CatBoost does include functionality for text features out of the box (something no other boosting library does), but we will not be using this feature as it would change the tokenization method of the training set for CatBoost which again would skew our comparison. Therefore, we will stick with our TFIDF-vectorization for CatBoost. (Update: There is an update at the bottom of this article showing how to use this feature and how it compares to our vanilla TFIDF-vectorization)</p><pre>cb = CatBoostClassifier(n_estimators=2000,<br>                        colsample_bylevel=0.06,<br>                        max_leaves=31,<br>                        subsample=0.67,<br>                        verbose=0,<br>                        thread_count=6,<br>                        random_state=1234)</pre><h4>Bonus: Histogram-based Gradient Boosting Machine (Scikit-Learn)</h4><p>Scikit-Learn recently released its own version of histogram boosting called HistGradientBoostingClassifierthat operates very similarly to LightGBM, offering speed many times greater than exact gradient boosting. However, one big caveat is that the Scikit-Learn implementation is only experimental as of March 2020, and only includes a handful of hyperparameters and two loss functions. One improvement of HistGradientBoostingClassifier over its exact counterpart is that it includes built-in support for missing values, with some simple rules for how to assign missing observations to leaf nodes.</p><pre>hgbm = HistGradientBoostingClassifier(max_iter=2000,<br>                                      validation_fraction=0.1,<br>                                      n_iter_no_change=15,<br>                                      verbose=0,<br>                                      random_state=1234)</pre><h3>Comparing our Competitors</h3><p>Time to perform our comparison! Before we start our training loop, we need to create a couple objects, such as putting our models into a list and extracting the class names for plotting later on:</p><pre>models = [dt, rf, ab, gbm, hgbm, xgb, lgbm, cb]<br>model_names = [i.__class__.__name__ for i in models]</pre><p>We also want to specify which of our models require early stopping within the .fit() method. Personally I prefer the convention of the non Scikit-Learn packages, where you pass a discrete validation set to the fit method for ES, rather than having the split performed internally (this gives you a bit more control).</p><pre>es_models = [&#39;XGBClassifier&#39;,<br>             &#39;LGBMClassifier&#39;,<br>             &#39;CatBoostClassifier&#39;]</pre><p>Finally, it’s time to train each of our candidate models! We are going to include a little bit of logic for the models that require a validation set to be passed in the .fit() method for early stopping. At the end of each run we are going to append our accuracy on the test set and how long it took to train each model:</p><pre>for m, n in zip(models, model_names):<br>    <br>    start_time = time()</pre><pre>    if n in es_models:<br>        m.fit(X_train_sub,<br>              y_train_sub,<br>              eval_set = [(X_valid, y_valid)],<br>              early_stopping_rounds=15,<br>              verbose=0)<br>    else:<br>        m.fit(X_train, y_train)<br>    <br>    run_time = time() - start_time<br>    accuracy = np.mean(m.predict(X_test) == y_test)<br>        <br>    df_results.loc[n] = [accuracy, run_time]<br>    <br>    del m</pre><h3>Checking Out the Results</h3><p>Now that we have stored all of our accuracies and run times, we are going to bundle everything in a data frame with our model names to make it easier to work with later. Finally, we are going to visualize our models’ testing accuracy and run times using matplotlib:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*27o3MrL_1FpcKv3GGfZDEQ.png" /><figcaption>Figure 1: Comparing test accuracy between our candidate models</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EidmWKDgy9LsF1cJFLGPwA.png" /><figcaption>Figure 2: Comparing run time (seconds) between our candidate models with absolute and relative times</figcaption></figure><p>Wow! As we can see from the plots above, it seems that LightGBM is the clear winner in this use-case, being over three tiLinearSVCmes faster than the next boosting algorithm while being the best in terms of test accuracy! LightGBM’s implementation is actually so fast that it can train the entire ensemble of trees faster than our single deep decision tree! While there may be some variation from run to run, this is such a large difference that we can say it is the winner with a fair degree of certainty.</p><p>Another takeaway from this is that CatBoost rather underperformed on speed, not quite matching the speed of XGBoost, an older library. However, it could be that with GPU-support enabled and some hyperparameter tuning this could change. Also surprising is the performance of Scikit-Learn’s HistGradientBoostingClassifier, which was considerably faster than both XGBoost and CatBoost, but didn’t seem to perform quite as well in terms of test accuracy. I expect that this boosting class will only continue to get better (remember it is experimental right now), as it didn’t even take advantage of feature subsetting and some of the other features present in other libraries.</p><p>Unsurprisingly it seems that AdaBoost and random forest both lagged quite a bit in run time, but I was rather disappointed in the performance of GradientBoostingClassifier. This class wasn’t meant for extremely large data, but this large of a gap in performance is interesting.</p><p>Well, it looks like we have our answer, LightGBM is the clear winner!</p><p><strong>But……</strong></p><h3>WAIT! What about other algorithms?</h3><p>I can already hear the data scientists and ML enthusiasts itching to jump to the comments section: “There are better models for bag-of-words!”, “You should compare these to other baseline models like neural nets or SVM!”, or “Boosting here doesn’t make sense!”.</p><p><strong>And to those having these thoughts, I completely agree with you.</strong></p><p>I wrote this article to see how boosting implementations would compare on a nonstandard problem and how they scale to wide data. In my opinion though, boosting algorithms are not ideal for this use-case, as you can get the same level of performance with much less compute time. Because we are dealing with extremely homogeneous data, we are going to compare our results to a much simpler alternative, one commonly used in text classification problems with bag-of-words: SVM. In particular, we are going to use Scikit-Learn’s LinearSVC class.</p><p>We are going to run our new model on the same data as our boosting candidates, recording our test accuracy and run time and adding them to the plots from earlier:</p><pre># Testing LinearSVC<br>svc = LinearSVC()<br>start_time = time()<br>svc.fit(X_train, y_train)<br>run_time = time() - start_time<br>preds = svc.predict(X_test)<br>score = np.mean(preds == y_test)</pre><pre>df_results.loc[&#39;LinearSVC&#39;] = [score, run_time]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Igwk3bp34mspXALJiHvWJQ.png" /><figcaption>Figure 3: Comparing the test accuracy of our SVM to our other models</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iPaWwKY1brZHCBL9eF9TxA.png" /><figcaption>Figure 4: Comparing the training time of SVM to our other models</figcaption></figure><p>The two plots above show exactly why you shouldn’t always jump to the most exciting and newest algorithms when approaching a problem that has an extensive body of prior literature. Support-vector machine, an algorithm that has been around for decades, performs nearly as well as our best state-of-the-art boosting model, <strong>while training nearly ten times faster! </strong>This just goes to show that tailoring your choice of model to the task at hand can be the difference between hours and minutes of train time!</p><h3>Conclusions</h3><p>We took a look at various boosting libraries and how they compare on extremely wide homogenous tabular data, ultimately finding that LightGBM was the fastest and most accurate of all that we tested. However, we also showed that more traditional algorithms such as SVM can still outperform the most advanced and cutting-edge methods! It will be interesting to see how the current set of popular boosting libraries evolve!</p><p>Thank you for reading! If you have any questions or comments feel free to leave them below!</p><h3>Update #1 — 4/28/2020</h3><p>The CatBoost team recently put out a new version, 0.23, which extends the internal text-vectorization functionality to CPU (previously you could only specify text features using GPU). I won’t get too deep into the details on how they vectorize under the hood (you can take a look at the official docs for that). Let’s also take a look at how this built-in vectorization does in comparison to our previous results: (Note that this is from a separate run so our results are slightly different than above, but the overall takeaways are the same)</p><pre># Testing the CatBoost text feature<br>model = CatBoostClassifier(n_estimators=2000,<br>                           text_features=[0],<br>                           colsample_bylevel=0.06,<br>                           max_leaves=31,<br>                           subsample=0.67,<br>                           verbose=0,<br>                           thread_count=6,<br>                           random_state=1234)<br>model.fit(X_train_text,<br>          y_train_sub,<br>          eval_set = [(X_valid_text, y_valid)],<br>          early_stopping_rounds=15,<br>          verbose=0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7-lUZtcSR-tnilDOf0rEA.png" /><figcaption>Figure 5: Comparing the test accuracy of the built-in text-vectorization of CatBoost</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7m_deWkakK0cI-pv_PJplw.png" /><figcaption>Figure 6: Comparing the training time of the built-in text-vectorization of CatBoost</figcaption></figure><p>As we can see from above, leveraging the internal text-vectorization of CatBoost nearly halved our train time! Furthermore, we can see that the new model generalizes better than any other method we tested so far. While we didn’t quite reach the same train speed as LightGBM, we did find that CatBoost with this newly-expanded feature can achieve pretty impressive accuracy!</p><h3>References</h3><ul><li>Large Movie Review Dataset<br><a href="https://ai.stanford.edu/~amaas/data/sentiment/">https://ai.stanford.edu/~amaas/data/sentiment/</a></li><li>Tutorial on TFIDF-Vectorization<br><a href="https://towardsdatascience.com/natural-language-processing-feature-engineering-using-tf-idf-e8b9d00e7e76">https://towardsdatascience.com/natural-language-processing-feature-engineering-using-tf-idf-e8b9d00e7e76</a></li><li>Introduction to Random Forest<br><a href="https://towardsdatascience.com/an-implementation-and-explanation-of-the-random-forest-in-python-77bf308a9b76">https://towardsdatascience.com/an-implementation-and-explanation-of-the-random-forest-in-python-77bf308a9b76</a></li><li>Helpful Post on Adaboost vs GBM:<br> <a href="https://stats.stackexchange.com/questions/164233/intuitive-explanations-of-differences-between-gradient-boosting-trees-gbm-ad">https://stats.stackexchange.com/questions/164233/intuitive-explanations-of-differences-between-gradient-boosting-trees-gbm-ad</a></li><li>Slides on Gradient Boosted Trees<br><a href="https://homes.cs.washington.edu/~tqchen/data/pdf/BoostedTree.pdf">https://homes.cs.washington.edu/~tqchen/data/pdf/BoostedTree.pdf</a></li><li>XGBoost Homepage<br> <a href="https://xgboost.readthedocs.io/en/latest/">https://xgboost.readthedocs.io/en/latest/</a></li><li>LightGBM Homepage<br><a href="https://lightgbm.readthedocs.io/en/latest/">https://lightgbm.readthedocs.io/en/latest/</a></li><li>CatBoost Homepage<br><a href="https://catboost.ai/">https://catboost.ai/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7c7f46fd956" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/boosting-showdown-scikit-learn-vs-xgboost-vs-lightgbm-vs-catboost-in-sentiment-classification-f7c7f46fd956">Boosting Showdown: Scikit-Learn vs XGBoost vs LightGBM vs CatBoost in Sentiment Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>