<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by josiah-kiok on Medium]]></title>
        <description><![CDATA[Stories by josiah-kiok on Medium]]></description>
        <link>https://medium.com/@josiah-kiok?source=rss-2cb61ea1a88d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*tKvp6MNReZheCwjd</url>
            <title>Stories by josiah-kiok on Medium</title>
            <link>https://medium.com/@josiah-kiok?source=rss-2cb61ea1a88d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 00:29:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@josiah-kiok/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beating Threes! with Reinforcement Learning]]></title>
            <link>https://medium.com/@josiah-kiok/beating-threes-with-reinforcement-learning-ae074dd28a68?source=rss-2cb61ea1a88d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae074dd28a68</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[games]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[pufferlib]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[josiah-kiok]]></dc:creator>
            <pubDate>Mon, 02 Mar 2026 19:03:42 GMT</pubDate>
            <atom:updated>2026-07-22T21:31:34.458Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the last month, I decided to start training a machine learning model to play Threes!, a sliding puzzle game, with pure reinforcement learning. This means the program doesn’t search or look ahead at all, which also means the model runs crazy fast. If you want to try it out for yourself, check out the link <a href="https://pseudonam-gc.github.io/threes-web/">here</a> (<a href="https://github.com/pseudonam-gc/threes-web">GitHub</a>)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P69y2dM-hM45VD3H26mriQ.gif" /></figure><p>It plays very well, being able to reach the challenging 6144-tile (an extremely difficult task) around a sixth of the time, and being able to reach the 3072-tile around 75% of the time.</p><p>It should be noted that I am relatively new to the world of RL, and very, very bad at Threes (my model would completely blow me out of the water). I studied my fair share of research papers and related projects, but as the <a href="https://reproducible.cs.princeton.edu/">replication crisis</a> can be definitely be felt in Machine Learning academia, not all information I have received may be fully accurate.</p><h3><strong>Table Of Contents</strong></h3><p>· <a href="#9dd4">Part 1: Background</a><br>· <a href="#0cfa">1.1: 2048 and Threes!</a><br>· <a href="#f31c">1.2: The World’s Hardest Game?</a><br>· <a href="#9d90">Part 2: On PufferLib</a><br>· <a href="#646b">2.1: Previous RL Attempts</a><br>· <a href="#6714">2.2: The Blowfish</a><br>· <a href="#15e1">Part 3: The Environment and Model</a><br>· <a href="#cc1a">3.1: The Environment</a><br>· <a href="#338f">3.2: The Reward Function</a><br>· <a href="#5d21">3.3: The Model</a><br>· <a href="#dd5f">Part 4: Training / Hyperparameter Optimizations</a><br>· <a href="#3815">4.1: Hyperparameters And You</a><br>· <a href="#d715">4.2: The Obligatory Learning Rate Tangent</a><br>· <a href="#8f56">4.3: The Other Ones</a><br>· <a href="#70b8">4.4: Hyperparameter Sweeps</a><br>· <a href="#edc9">Part 5: Results</a><br>· <a href="#d316">5.1: Initial Runs</a><br>· <a href="#4528">5.2: Final Runs</a><br>· <a href="#7da6">Part 6: Conclusion</a></p><h3>Part 1: Background</h3><h3>1.1: 2048 and Threes!</h3><p>Hundreds of millions of players would recognize 2048, the hit indie game created by Gabriele Cirulli over ten years ago. The game is extremely simplistic: players would slide around powers of two on a 4x4 grid, equal tiles would combine (2 and 2 make 4, 4 and 4 make 8, and so on.). Players would try to create the largest tiles possible in order to achieve the 2048 tile (or <em>much, much</em> higher) before the board was filled and they were left unable to move.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*zq7IkjORYCm_Oz2CK905ZA.png" /><figcaption>In practice, 131072 is basically unreachable, but the image is just for example.</figcaption></figure><p>Do you notice any similarities in the following image?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*BU-oJ_M9ppQ36HDEDVJGiA.png" /><figcaption>Image from LogicCube, the #1 World Record holder who broke his own record a few days ago.</figcaption></figure><p>Much lesser known was the game that inspired it, Threes!, likely due to it being harder right out of the gate. The rules of Threes! are mostly the same as 2048 — or maybe it’s the other way around, with a few key differences.</p><ol><li>Tiles start at 1, 2, 3, 6, 12… (then continuing to double) instead of 2, 4, 8, 16, 32.</li><li>Two different digits (1 and 2) are required to make 3.</li><li>Tiles only slide one space when moved.</li><li>Players get information about the next tile to spawn, with an equal probability of being a 1, a 2, or a 3, and a rarer chance of being an unknown bonus high-value tile.</li><li>There are also far more intricacies to the spawning mechanism that can be utilized by the player, but they are too verbose to discuss here.</li></ol><p>By far, the most interesting gameplay difference is the lowest merge: instead of combining 2 and 2 to make 4, players must instead combine 1 and 2 to make 3. Marginal enough, right?</p><p>The problem is, instead of being able to reliably merge any existing 2s with a future 2 (which 2048 will almost always provide you with), you might get a lot of 2s that clutter the board without any 1s, or vice versa! The game does have a system in place to ensure that the difference between 1s and 2s is at most four, but this means players can lose up to four tiles of space instead of one!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*m2n3qAL7sJ9FI0AFMWJAKg.png" /><figcaption>From Discord user ekisacik, who received too many twos to be able to recover. Too bad.</figcaption></figure><p>The one and only compensation is that the locations where tiles spawn are more predictable and can be influenced by clever play. However, unless you give it special attention, new tiles tend to spawn in the worst possible positions, particularly when you move your big tile out of a corner!</p><p>The game designer of Threes, Asher Vollmer, spent months on this trying to ensure a simple algorithm couldn’t easily beat the game, unlike pre-endgame 2048. He succeeded.</p><h3>1.2: The World’s Hardest Game?</h3><p>The two hardest Threes! achievements (not counting the uncompleted community achievements) are as follows:</p><ol><li>Create a 6144-tile. The vast majority of players haven’t gotten close, and the best players can only reach it around half the time. (The equivalent in the game 2048 is 16384.)</li><li>Create a 12288-tile. 12288 is the final tile which can be mathematically reached, even with absolutely perfect play, barring extreme RNG manipulation. (The 2048 equivalent is 32768.) Regardless, the game automatically ends once this tile is created.</li></ol><p>So, how difficult is it to achieve these?</p><p>Let’s look at the Challenge Enthusiasts site, which compiles difficult achievements from some 2,000 single-player games and tiers them. It’s not perfect, as many of these games come from extremely different genres, but it should give some sense of comparison. Higher-numbered tiers are more difficult.</p><p>The hardest Dark Souls game is Tier 2, also shared with DELTARUNE, which has an achievement for no-hitting the Chapter 3 final boss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*y4NHi6pER5FJClrE3tVDgw.png" /><figcaption>Bet you don’t often see Dark Souls be the easier <em>game in a comparison.</em></figcaption></figure><p>Touhou is a shoot-em-up genre where you have to dodge hundreds, if not thousands, of bullets — clearing such a game is typically around Tiers 3–4. Games in the series without a perfect run achievement are only Tier 3. However, there are multiple fangames with comparable difficulty that <em>do</em> require a perfect run for completion around Tier 4. While I could argue why I think those are underrated, the following picture should get the point across.</p><figure><img alt="UNR, the fourth hardest attack in Subterranean Animism." src="https://cdn-images-1.medium.com/max/682/1*3ft5EHpszpYad2jf-LDbCg.png" /><figcaption>Yes, the pattern is RNG vomit. Best hope you don’t get walled, now!</figcaption></figure><p>Celeste’s Farewell Golden is in Tier 5. Below is one of the easier sections, which I have cherry-picked because it looks scary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*xb4Llv8oxuiCk38sLPKYoQ.png" /><figcaption>Oh, the electric barriers move, by the way.</figcaption></figure><p>Who would win? 20–40 minutes of flawless dodging or platforming with no mistakes, or the cute minimalistic sliding numbers game?</p><p>The answer is, of course, The Cute Minimalistic Sliding Numbers Game, which is <a href="https://en.wikipedia.org/wiki/Brain_rot">Tier 6 out of 7</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*QHZwdPexAHKkY9uZt3iYgQ.png" /></figure><p>Reaching 6144 is quite difficult already, as there are less than one hundred confirmed clears. Most players don’t get anywhere close.</p><p>Less than ten people have achieved a confirmed 12288, and that’s not for lack of players — even the best players have a ~5% success rate, or even less. There’s even a community strategy guide dedicated to reaching 6144, which has close to zero information on 12288.</p><p><a href="https://docs.google.com/document/d/1Ra8YbOMz4fMTwAPgeI2Y1_-q7yO1I0XDvAmv2_KKtZk/edit?tab=t.0">Threes! — Updated Steam Guide (Draft)</a></p><p>Let’s see if RL can match them.</p><h3>Part 2: On PufferLib</h3><h3>2.1: Previous RL Attempts</h3><p>While I had previously attempted to design my own RL environment for Connect 4 in a university project, that project ended in disaster (an agent that wouldn’t block wins-in-one about a fourth of the time!) for a few reasons.</p><ol><li>Deep Q-Networks performed poorly on complex two player games. Representing a deep minimax game tree, even a simple one, is substantially harder than the Atari games of old which DQN performed well on. When I switched to using stable-baseline’s PPO wrapper, I instantly got good enough results that could take games off intermediate players like myself if they got complacent.</li><li>Running and simulating games took forever. We were using Python, and as everyone knows, unoptimized Python is like fifty times slower than C++. As is typical of machine learning, vastly more time was spent tweaking hyperparameters and debugging than actually training the final model, and this made the programming process very painful.</li><li>Implementing everything from scratch (including the environment, DQN, the Replay Buffer, etc.) made for extremely difficult debugging.</li></ol><p>It’s not like the model learned nothing; it knew to prioritize the center, and it would spot wins in one enough of the time. But it was constantly outperformed by a simple, 2-depth heuristic search.</p><h3>2.2: The Blowfish</h3><p>With a few more years of CS experience under my belt, I decided I was going to give RL another shot. A friend recommended PufferLib, an open-source library that uses PPO. PufferLib provides the following:</p><ul><li>Fast vectorized environments. Running environments in parallel is much more cache efficient. Some projects even demonstrate a 10x speed increase, though PufferLib has many other optimizations that make it far faster.</li><li>Simplified environment training. The train loop is just a 1000-line script with minimal Object-Oriented structures. It’s not necessary to understand most of the codebase to use or process it.</li><li>Automatic hyperparameter sweeps using PROTEIN (an improved variant of CARBS). The assurance that a set of hyperparameters are close enough to optimal is such a burden-reliever, as you don’t have to manually train some twenty models to confirm that gamma is 0.995 or 0.999 (this can make quite a difference!).</li></ul><p>Most importantly, it has achieved prior success in many environments, many of which can be utterly complex. It even beat <a href="https://drubinstein.github.io/pokerl/">Pokemon Red</a>, a comparatively massive undertaking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*MtVpLsfbhbuh34UysvP1KQ.png" /></figure><p>Apparently, the LLM Gemini <a href="https://arstechnica.com/ai/2025/05/why-google-geminis-pokemon-success-isnt-all-its-cracked-up-to-be/">beat Pokemon Red</a> shortly after David Rubinstein and co. did with PufferLib. Despite having internet access to all Pokemon statistics, map information, and a wealth of strategy guides, it still required a lot more handholding. Another win for RL!</p><p>It didn’t take much persuasion for me to switch to PufferLib after reading that. I started toying with Target, a baseline environment featuring an agent moving along a large grid onto a randomly placed green cell. I added twenty red cells (also randomly placed) which penalized the agent if it touched them, and the model learned to avoid them surprisingly well!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*nefM8uOPXrVp30v4jor-Mg.gif" /><figcaption>Yes, this is a trivial search problem, but we’re using Pure RL instead.</figcaption></figure><p>So what’s the next step? Well, there was already a 2048 environment I could work off of, so why not see how well a model could utilize the additional information and constraints that Threes! provides to survive the much harsher early and middlegame?</p><h3>Part 3: The Environment and Model</h3><h3>3.1: The Environment</h3><p>Creating the environment was pretty smooth sailing for the most part, once I had learned the rules for all the spawning and movement mechanics. There were a few options I could have chosen, though:</p><p>First, I initially used one-hot encoding for the board, meaning the model perceives each different piece value as an entirely different object rather than ‘three times the previous piece’. Otherwise, the model would likely learn a false numerical relationship between pieces, and would especially struggle to handle the fact that some numbers were <em>way higher</em> than others. A <em>linear model</em> with a few activation functions would be required to distinguish 6144 and 3072, but also 6 and 3.</p><p>I later replaced this with embeddings, with dimension size 8. However, both functioned similarly enough and neither were the main performance bottleneck: that would be the large fully connected layers.</p><h3>3.2: The Reward Function</h3><p>Secondly, the model’s reward function had to be well-chosen. The reward function is the only factor the model uses to determine whether how successful a game was — and how it would learn. The primary concern was that Threes was an exponential game, in that scores were often sums of values of the form 3^n.</p><p>Neural networks are often piecewise-ish combinations of linear layers, so they do not learn exponential functions very well. It might as well try modelling y=e^x with three line segments. You’ll find that models would often resort to casework to get around this problem. So, it was in my interest to choose a reward function that didn’t exponentially explode.</p><p>There were quite a few other options for the reward function that weren’t exponential:</p><ol><li>Game length. The longer you survive, the cleaner your board must have been to survive.</li><li>Number of merges. Not only does this have direct correlation with game length, but it incentivizes merging large tiles together.</li></ol><p>The authors of the 2048 environment had already chosen a decent reward function, which was the number of merges, multiplied by how many ‘layers’ deep the piece was. Given their success, I saw no reason not to use it as well.</p><p>The final score in Threes! is also entirely computed at the end, though the model never sees it and it’s only useful as a human evaluation metric. I ended up breaking it up into tinier pieces that added score upon every merge. However, in the end, the final score would be equal to the actual score.</p><h3>3.3: The Model</h3><p>Third, the model itself. I didn’t deviate much from the 2048 structure outside of layer sizes, as it passed most of my checks for what a decent actor critic network should look like. It looks like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*HXp0ETuvDHmQfQq9CbVW7A.png" /></figure><p>An AC network has two parts, an Actor and a Critic. Both receive the state of the board, The Critic assigns an approximated value to the board position, and the Actor chooses the best move in the current position, using the Critic’s estimated values to improve its behavior. Note that the Critic’s value is different from the reward function, as it tries to estimate <em>the sum of all future rewards</em> instead of just the current or final reward value. We simply do not have access to this, so we need some function to estimate this.</p><p>Empirically, keeping the Actor and Critic somewhat separate has been shown to give better results, as the task of evaluating the current board state and choosing the right action are different enough. While they’re not fully separate here, they have two separate FC layers which allows them to specialize.</p><p>The LSTM layer allows the model to learn relationships between its state and its previous states. It also adds minimal computation overhead. While there technically is zero benefit to knowing board history, it could allow the network to learn the effects of each action, as they progress the board from its current to its next state. There is, however, an additional cost to the LSTM layer, which will be covered later.</p><h3>Part 4: Training / Hyperparameter Optimizations</h3><h3>4.1: Hyperparameters And You</h3><p>Enough following the 2048 architecture. Training, unfortunately, does not spare me the luxury of following along and deviating from the template whenever necessary.</p><p>Pufferlib’s implemented PPO variant has <em>far too many</em> hyperparameters, probably some 25–40 in total. If I had a mortal enemy who could adjust just one of these numbers to screw with me, the results vary from ‘this model now takes twice as long to train’ to ‘the model is lobotomized and incapable of learning anything’, with the majority of situations being the latter.</p><p>Apparently, even those studying RL are not immune to brainrot. This sus number of timesteps was part of the default parameters from 2048.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*FKDvjom6e_01mxYB8G-FDA.png" /></figure><h3>4.2: The Obligatory Learning Rate Tangent</h3><p>The primary hyperparameters that I found worth adjusting were the following:</p><p>1 — Learning rate (LR). Basically, imagine the model as a ball rolling around a mountain ridge, and where we’re trying to bring it to the lowest valley. (It even has some form of momentum, thanks to the Adam optimizer). If there’s a slope, it’ll take the quickest path down. Learning rate is the speed at which it descends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*MBhgtIKeltnZcxyhwE236w.png" /><figcaption>Source: Stanford CS231</figcaption></figure><p>If the LR is too low, the ball would take forever to go anywhere. For instance, if LR is ten times lower, performance would be theoretically close to ten times slower.</p><p>If the LR is too high, the ball would overshoot the hole. If the LR rate is <em>really</em> high, the ball flies off into the stratosphere. Obviously, this is terrible.</p><p>This creates what I affectionately refer to as NaN hell, where the model is effectively dead and all your aforementioned training is wasted.</p><p>As the game gets harder, the ball must fit through more and more precise gaps, so LR must be lowered accordingly. This (among other stability reasons) is why learning rate annealing is all but required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*Iixa_ezdnY-d1GHf5sptaw.png" /><figcaption>ResNet, which demonstrated substantial improvements from learning rate reductions mid-training. I also observed this myself while training.</figcaption></figure><p>Additionally, there are many ‘ledges’, or local minimums where the model can stop at. If our learning rate or PPO clipping values are too low, the model could end up permanently at a local minimum instead of near the bottom.</p><h3>4.3: The Other Ones</h3><p>2 — Gamma. I said earlier that the reward function is a sum of future rewards. This isn’t entirely true; the model weights rewards further in the future higher whenever gamma is high, and vice versa. Too low of a gamma will create a ‘greedy’ model that will not think long term, and too high of a gamma leads to overfitting and other consequences. It’s basically just a geometric series, where the reward n rounds from now gets multiplied by gamma raised to the nth power.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*ESTpWcEMq55Yu6du_mCatQ.png" /></figure><p>3 — Total timesteps. This is how many steps the model trains for. Not particularly surprising.</p><p>4 — Model size (input_size and hidden_size). The right-sized model can theoretically learn more, and do so faster. Too large, and it’s too slow for both learning and execution. Too small, and it won’t learn anything beyond the basics, or anything at all. I found that the model can play decently well with a size of 256, but I used a size of 1024 to push the success rate higher.</p><p>I did around 150 experiments adjusting these values under different conditions. Some of my observations (as well as graphs) are in the latter section.</p><h3>4.4: Hyperparameter Sweeps</h3><p>As I’ve elaborated on before, it is likely that far more time is spent setting hyperparameters or dealing with issues than training the final model. After all, the difference between good hyperparameters and passable ones could mean the difference between success and failure.</p><p>Unfortunately, to my limited knowledge, there is no way to determine which hyperparameters are decent without some amount of trial and error, and this includes training a whole slew of models ranging from effective to useless. For each new model we train, we have to launch another training run from start to finish.</p><p>The old-fashioned technique for hyperparameter sweeps is called grid search. If there are more than 3–4 hyperparameters, you freeze the excess at reasonable, specific values — which unfortunately requires RL experience and domain knowledge. Then, you assign each hyperparameter you are adjusting a list of potential values — which also requires RL experience and domain knowledge! Then, you iterate through all possibilities, and the list grows exponentially large for each hyperparameter you run through the sweep. But remember, if any of your other hyperparameters are set too far incorrectly, training might be near-impossible!</p><p>Imagine you’re testing 5 values for just learning rate, gamma, and hidden size. That’s 125 models, and odds are, 100 of them are going to obviously suck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I-30P2wJ42liKLtvkMPiDA.png" /><figcaption>Wikipedia example, where blue regions are good, and orange regions are bad. With grid search, we’d spend so much time checking the garbage areas rather than climbing the blue hills.</figcaption></figure><p>There’s a few improvements we can do, of course. The first is simply killing off runs that are clearly dead in the water before they can fully complete. The second is using PufferLib’s PROTEIN sweeps, which uses a variant of local search to find the Pareto front.</p><p>What this means is that PROTEIN attempts to find the ideal model parameters for various possible training durations (or other costs) by trying individual points. Later on, it focuses more on points that perform better, meaning that we can afford to have much more hyperparameters being swept, which is crucial for PPO. PufferLib projects can have up to 30–50 hyperparameters, which would be completely impossible to grid search!</p><p>Even still, training like this would be a substantially difficult task that would take hours to train, and I didn’t have the compute to multiply that for each sweep. Instead, I stopped each sweep as they approached 5–10 minutes of runtime, allowing me to get around a hundred trial runs in. Of course, the hyperparameters that perform well at the start will give performance nowhere near the hyperparameters that perform best in the long run, but it’s at least something — and I can always drop the learning rate later.</p><h3>Part 5: Results</h3><h3>5.1: Initial Runs</h3><p>The goal of the project was to get as close to the 6144-tile as possible, so I judged performance by the size of the largest tile, divided by 6144. While consistent 6144s are not realistically possible, I would be happy if I could be fairly consistent reaching 3072 and hit 6144 every now and then.</p><p>The initial results after training for 500 million steps with LR=0.04 looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vm-rDi0fyJDiWZgPQd1crQ.png" /><figcaption>The solid line is a running average across the nearest 200 episodes; the jagged line is the average return per iteration. Note that 4096 games are running in parallel per iteration, which mitigates a lot of the noise.</figcaption></figure><p>The model was often bouncing between max tiles of 786 and 1536 in the first iteration, which was already pretty impressive. However, it would quickly plateau. I would resume from a checkpoint and then halve the learning rate, which allowed it to go further but at decreased progress. I would repeat this every time I saw the model plateau, and the model would advance a bit further every time.</p><p>Midway through training, I wondered what would happen if I ran another hyperparameter sweep, except centered around fine-tuning a checkpoint model. My main speculation was that the model would ‘freeze itself’ at first to avoid messing itself up, but I was hoping that it would learn something later in the process.</p><p>This little side tangent used quite a bunch of compute time just for the chosen hyperparameters to provide worse results than the first ones, though it was still learning — albeit very slowly.</p><p>In pink, the model with the initial hyperparameters. In brown, the model with the later set. Initial ones performed marginally better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bx_JAt8be1K3CRevxO94YQ.png" /></figure><h3>5.2: Final Runs</h3><p>As for the final model, I was able to reach a final performance of 0.55, meaning it was able to make 3072-tiles most of the time! It could also make 6144-tiles with decent frequency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*M1DQCURDYbnCSRa8R-zMGw.png" /></figure><p>==================================================<br>Results from 200 games:<br>==================================================<br>Highest Tile Count Percentage<br>--------------------------------------------------<br>6144 33 16.5%<br>3072 129 64.5%<br>1536 32 16.0%<br>768 6 3.0%<br>==================================================</p><p>And it was able to hit 12288-tiles in training! This is sufficiently rare that I do not have video of it, but this occurs with probability 0.065% (roughly 1 in 1500 times).</p><h3>Part 6: Conclusion</h3><p>The model was a success, playing Threes better than all but the best human players (again, only some hundred players have confirmed 6144-tiles)!</p><p>I didn’t really have the expectation of beating the best human. Nor was I beating the best Threes AI, which got a 6144-rate of 20%, which used a form of search called Expectimax. (ERRATA: This was an early estimate. halfrost’s actual rate was 14%, according to one of their GitHub logs, making my model barely edge theirs out!)</p><p>But my model was much, much faster. And with its speed, it was able to learn to be able to hit 12288 a fair amount of the time, at least in training runs, which no other AI to my knowledge has done.</p><p>I also learned a lot from the process, getting more familiarity with each of the following:</p><ul><li>The PufferLib library, of course, as well as NumPy/Torch (more prevalent when running experiments with ConvNets and Embeddings).</li><li>C Programming — PufferLib uses C-wrappers in order to make training as fast as it is. While I have been fairly acclimated to C++ for a while, it’s been a while since I’ve done anything without the C++ standard library.</li></ul><p>My main takeaways were:</p><ul><li><strong>Vectorization and running numerous environments in parallel is absolutely critical in RL.</strong> Models take a lot more iterations to learn than we do. I do find it impressive that humans are a lot better at learning with <em>far</em> less experience.</li><li><strong>If all else is equal, simple models are the best</strong>, as you have less moving parts to finagle and spend time and money fixing. I learned this while briefly toying with CNNs, as well as cyclical learning rate annealing. While I had good reason to believe they would perform better, they only barely matched the performance of the models without them while creating additional complexity. Simple is also a lot easier to implement.</li><li><strong>Deal with the tightest bottleneck first, and avoid premature optimization.</strong> Prematurely optimizing other things could always be done later. I learned this the hard way while trying to make my environment as fast as possible — the bottleneck, I later learned, was the model, and what I did had little effect. Later, I tried to decrease the input size of the board, because I was afraid that embeddings or one-hot encodings were too large: I had entirely forgotten that the fully connected parts of the model were much larger and therefore more time consuming. Better to just focus down the narrowest chokepoint first, and then adjust things later in the process.</li></ul><p>There’s also a bit more to be done, were I to return to this in the future. I could attempt model distillation in order to allow for faster evaluations and faster sweeps, since I speculate most of the massive parameter count is excessive. I also theorize I could use my model (with a bit of fine-tuning) with Monte-Carlo Tree Search, which would almost certainly improve the success rate drastically at the cost of time. I have always wondered what the optimal success rates are, and maybe I’ll be able to get very, very close to them with MCTS!</p><p>Update: I have found out that halfrost’s 6144-rate of 20% was an estimate, and achieved a 14% rate under their most recent sample of 50 games. I also ran my model 10,000 times as opposed to the previous 200, achieving a much more promising success rate of 21.5% and a 3072-rate of 77.7%! Below is a comparison to every other research paper and model I was able to find.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kjUgghOxpry8OHsJafjT2w.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae074dd28a68" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>