I Built Three Reinforcement Learning Agents. The “Improved” Ones Lost.
What five random seeds taught me about variance, baselines, and why a higher average return can still mean a worse algorithm
Most reinforcement learning tutorials present algorithms in a clean progression. REINFORCE, then REINFORCE with a baseline, then actor-critic methods like A3C. Each one framed as strictly better than the last, backed by a proof that the baseline doesn’t bias the gradient and only reduces its variance.
For my Advanced AI module at the University of Moratuwa, I built and trained all three, on the same task, with the same network, the same learning rate, the same five random seeds. The theoretically “improved” versions lost to the simple one.
This article covers what I built, what actually happened, and why the result says more about experimental design than about any of the three algorithms.
The Task
The environment is InvertedPendulum-v4, a continuous control task from Gymnasium built on the MuJoCo physics engine. A cart on a track has to balance a pole by applying continuous force, left or right. The agent gets +1 reward per timestep the pole stays upright, capped at 1000 steps. A perfect episode scores 1000.
Three algorithms, identical setup across all of them:
- REINFORCE — the baseline algorithm, Monte Carlo policy gradient with no variance correction
- REINFORCE with a learned baseline — a critic head predicts expected return, the policy update uses the advantage instead of the raw return
- A3C — four parallel worker processes, each running its own environment, pushing gradients into one shared network with no locking
Each algorithm trained for 5000 episodes across five seeds (1, 2, 3, 5, 8). Fifteen full training runs.
REINFORCE: High Variance, High Ceiling
REINFORCE has no critic and no variance correction. Run an episode, compute the discounted return at every step, update the policy once using that return directly as the learning signal.
Across five seeds, the results diverged sharply:
- Best seed: 795.34 mean return over the last 100 episodes, close to the 1000-step ceiling
- Worst seed: 203.16
- Standard deviation across seeds: 280.44
Same code, same hyperparameters, only the random seed changed. One seed also took roughly 30 minutes to train, against 8 to 15 minutes for the others. Once a policy learns to balance well, its episodes get longer, and longer episodes take proportionally longer to simulate and backpropagate through. Training time turned out to be a downstream signal of policy quality, not just a resource constraint.
REINFORCE has no mechanism to control this. Every episode’s return goes directly into the update, unfiltered.
Adding a Baseline: Lower Variance, Lower Ceiling
The standard fix for REINFORCE’s variance is a learned baseline. Instead of pushing the policy using the raw return, subtract what the model expected to get from that state. What remains is the advantage: did this outcome beat expectations or not.
I added a second output head to the network, a critic trained with mean squared error against the actual return. Subtracting a state-dependent baseline like this is provably unbiased. It should only reduce the variance of the gradient estimate, not change its expected direction.
Results across the same five seeds:
- Mean return, last 100 episodes: 25.73
- Standard deviation across seeds: 1.2
The standard deviation dropped from 280 to 1.2. The variance reduction worked exactly as the theory predicts.
But the mean return dropped just as sharply, from 554.65 down to 25.73.
The critic shares the same tiny learning rate as the policy network and most of the same layers. Early in training its value estimates are inaccurate, so the advantage signal it produces is still noisy, just noisy in a different way than the raw return was. That noise appears to be damping the larger, riskier policy updates that let REINFORCE occasionally find a good solution on its better seeds. A baseline reduced variance. It did not improve performance. Those are not the same outcome, and this project was the first place I saw that distinction show up concretely rather than as a caveat in a textbook.
Building A3C
A3C required actual engineering, not just a change to the loss function.
Four worker processes run in parallel, each with its own environment and its own local copy of the network. Each worker computes gradients over short rollouts and pushes them into one shared network with no locking between workers, a design known as Hogwild-style asynchronous updates.
Get Pathum Dilshan’s stories in your inbox
Join Medium for free to get updates from this writer.
What I built:
- Four worker processes, each running InvertedPendulum-v4 independently
- One shared network with a shared Adam optimiser, its momentum buffers held in shared memory so every worker process reads and writes the same optimiser state
- 20-step rollouts per update, with an entropy bonus to keep exploration active
- No locking. Workers push gradients and pull updated weights without waiting on each other
One implementation detail broke on a platform switch. I developed this using Python’s fork method for spawning processes, standard on Linux. Testing on Windows failed immediately, fork does not exist there, only spawn does. A one-line fix, but a clear reminder that multiprocessing code is far less portable than it looks.
Results:
- Mean return, last 100 episodes: 35.7
- Standard deviation across seeds: 5.21
A3C landed slightly ahead of the baseline method, consistent with four workers collecting more varied experience in parallel. It also completed all five seeds in under seven minutes combined, against well over an hour for REINFORCE. Part of that speed came from A3C never escaping its own low-return range, its episodes stayed short throughout training. Fast training here was not a good sign. It was the same underlying problem as the baseline method, just faster to observe.
What the Numbers Actually Say
AlgorithmMean return (last 100 ep)Std across seedsBest seedWorst seedREINFORCE554.65280.44795.34203.16REINFORCE + Baseline25.731.227.224.38A3C35.75.2141.9328.55
Reporting only the mean makes REINFORCE the clear winner. Reporting only the standard deviation makes it the clear loser. Neither view alone is sufficient, and either one in isolation would have let me reach whatever conclusion I wanted going in.
The honest read: REINFORCE’s mean is being carried by one or two seeds that happened to find a working policy, not by five seeds converging to similar behavior. The baseline and A3C methods are doing exactly what they are designed to do, reducing variance across runs, but with hyperparameters that were not tuned to also chase a higher return, that stability came at a real cost to the ceiling either method could reach.
What I Learned
Seed count changes conclusions, not just confidence intervals.
Five seeds were enough to make REINFORCE look dramatically better or dramatically worse than the other two methods depending on which single statistic I reported. A sixth seed could plausibly flip which algorithm looks best on the mean alone.
A shared learning rate for actor and critic is a real design decision, not a neutral default.
Using the tutorial’s original learning rate for the critic head, rather than tuning it separately, is the most likely reason both extensions plateaued below REINFORCE’s peak. The critic’s inaccurate early value estimates were feeding a noisy advantage signal into the policy update.
Cross-platform multiprocessing bugs surface exactly when you don’t want them to.
The fork-versus-spawn issue only appeared when moving from development to a different machine, well after the core algorithm was working and tested.
Training speed can be a symptom, not a feature.
A3C finishing five seeds in minutes looked like an efficiency win until I connected it to the fact that its episodes never got long enough to be expensive, meaning it never escaped the low-return regime the baseline method was also stuck in.
What This Means
None of this means baselines or actor-critic methods are worse than REINFORCE in general. It means that comparing algorithms on a handful of seeds, with hyperparameters shared across all of them rather than tuned per method, produces a comparison that is fragile in a specific, identifiable way.
The next step worth testing is giving the critic in both the baseline and A3C versions an independently tuned, faster learning rate, separate from the actor, to see whether the variance reduction can be kept without giving up the higher returns REINFORCE occasionally reaches. That is the open question this project left me with.
Technical Details
Environment: InvertedPendulum-v4 (MuJoCo, via Gymnasium)
Seeds: 1, 2, 3, 5, 8–5000 episodes per seed per algorithm, 15 total training runs
Network: shared 16–32 unit MLP trunk (Tanh activations), identical across all three methods
Optimizer: AdamW / Adam, learning rate 1e-4, discount factor gamma = 0.99
A3C specifics: 4 worker processes, 20-step rollouts, entropy coefficient 1e-3, value loss coefficient 0.5, gradient norm clipping at 40
Code: Python, PyTorch, Gymnasium, native multiprocessing (torch.multiprocessing with a shared-memory Adam optimizer)
References
Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256.
Mnih, V. et al. (2016). Asynchronous methods for deep reinforcement learning. ICML 2016.
Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.
This project was completed as part of the module In26-S2-CS5801 Advanced Artificial Intelligence, MSc in Data Science and AI, University of Moratuwa.
Pathum Dilshan is an MSc student in Data Science and AI at the University of Moratuwa and a Network Planning and Optimization Engineer at Mobitel Sri Lanka. He writes about applied machine learning and AI engineering.
