A Practical Guide to Double DQN:
Reducing Overestimation in Deep
Reinforcement Learning
Reinforcement learning, also known as RL, is a method to train an agent to make sequential decisions. This is done through allowing the agent to interact with an environment and gradually receive feedback in the form of rewards. This can be seen in many real-world cases, including robotics, medical decision making, and dynamic resource allocation. [1]
Classical RL algorithms such as Q-learning learn a value function that tells the agent how good it is to take a certain action in a given state. However, these methods historically struggled with environments that have large or continuous state spaces, where it is not feasible to maintain a table of all state–action values.
Deep Q-Networks (DQN) addressed this limitation by combining Q-learning with deep neural networks, enabling agents to learn directly from high-dimensional inputs such as images [2]. DQN was a major breakthrough in RL and demonstrated superhuman performance on Atari games. However, despite its successes, DQN suffers from a known issue: overestimation bias, where the value of an action is frequently overpredicted. This can lead to unstable learning, inefficient exploration, and suboptimal policies.
Double DQN (DDQN) was proposed as a simple yet powerful solution to this issue. By decoupling the action selection and action evaluation steps used during value updates, DDQN significantly improves learning stability and reduces overestimation — often leading to faster convergence and better overall performance. [3]
In this tutorial, we explore how Double DQN works, how it differs from standard DQN, and how to implement and visualise its behaviour in practice.
Theoretical Explanation
Q-Learning and DQN
In standard Q-learning, we maintain a value function Q(s, a) that estimates the expected return of the action a in state s. The update rule is:
DQN replaces Q-table with a neural network, enabling value estimation in high-dimensional spaces. To stabilise training, DQN introduces two operators: experience replay and the target network.
Experience replay stores transition states and actions (s, a, s’, a’) in a buffer and samples mini batches to break correlation. The target network is a periodically updated copy of the main network (denoted Q(θ −)). This is used to compute target values. [4]
The DQN target is:
The loss is:
The problem with DQN: Overestimation
This method involves a max over estimated Q-values, which contain noise. When this happens, the max tends to select overestimated values. This makes the target overly optimistic, especially early in training, leading to divergence or unstable policies. [5]
The solution: Double DQN
Double DQN (DDQN) was introduced as the solution, mainly by splitting the roles of the two operators. Action selection now uses the online network only, and action evaluation now uses the target network only. Thus, the DDQN target is [3]:
This subtle change prevents the target value from being inflated by the same network that chooses the action. As a result, Q-values become more realistic, training stabilises, and the agent learns more reliable policies.
Code Demonstration
Below is a concise PyTorch-style snippet illustrating the core Double DQN update. It highlights the action-selection/action-evaluation decoupling, which is the main innovation of DDQN.
To evaluate Double DQN VS DQN, I trained an agent on CartPole-v1, a classic control problem where the goal is to balance a pole on a moving cart. It is lightweight and ideal for RL algorithm demonstrations. This is done via the code below:
# Training function
def train_agent(
env_id="CartPole-v1",
num_episodes=5000,
gamma=0.99,
batch_size=64,
buffer_capacity=50_000,
lr=1e-3,
target_update_freq=1000,
min_buffer_size=1000,
double_dqn=False,
):
env = gym.make(env_id)
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
online_net = QNetwork(state_dim, action_dim).to(device)
target_net = QNetwork(state_dim, action_dim).to(device)
target_net.load_state_dict(online_net.state_dict())
target_net.eval()
optimizer = optim.Adam(online_net.parameters(), lr=lr)
replay_buffer = ReplayBuffer(buffer_capacity)
epsilon_start = 1.0
epsilon_final = 0.02
epsilon_decay = 30_000 # in steps
global_step = 0
episode_rewards = []
max_steps_per_episode = 500
for episode in range(num_episodes):
state, info = env.reset()
episode_reward = 0
for step in range(max_steps_per_episode):
global_step += 1
# Epsilon-greedy policy
epsilon = epsilon_final + (epsilon_start - epsilon_final) * np.exp(
-1.0 * global_step / epsilon_decay
)
if random.random() < epsilon:
action = env.action_space.sample()
else:
with torch.no_grad():
s = torch.tensor(state, dtype=torch.float32, device=device).unsqueeze(0)
q_values = online_net(s)
action = int(torch.argmax(q_values, dim=1).item())
next_state, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated
replay_buffer.push(state, action, reward, next_state, done)
state = next_state
episode_reward += reward
# Update step
if len(replay_buffer) >= min_buffer_size:
(
states,
actions,
rewards,
next_states,
dones,
) = replay_buffer.sample(batch_size)
states = torch.tensor(states, dtype=torch.float32, device=device)
actions = torch.tensor(actions, dtype=torch.long, device=device)
rewards = torch.tensor(rewards, dtype=torch.float32, device=device)
next_states = torch.tensor(next_states, dtype=torch.float32, device=device)
dones = torch.tensor(dones, dtype=torch.float32, device=device)
# Current Q-values: Q(s,a)
q_values = online_net(states)
q_sa = q_values.gather(1, actions.unsqueeze(1)).squeeze(1)
# Target Q-values
with torch.no_grad():
if double_dqn:
# DOUBLE DQN LOGIC
# 1 - action selection using online network
next_q_online = online_net(next_states)
next_actions = torch.argmax(next_q_online, dim=1, keepdim=True)
# 2 - action evaluation using target network
next_q_target = target_net(next_states)
next_q = next_q_target.gather(1, next_actions).squeeze(1)
else:
# STANDARD DQN LOGIC
next_q_target = target_net(next_states)
next_q, _ = next_q_target.max(dim=1)
targets = rewards + gamma * next_q * (1.0 - dones)
# Loss and optimisation
loss = nn.MSELoss()(q_sa, targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Periodically update target network
if global_step % target_update_freq == 0:
target_net.load_state_dict(online_net.state_dict())
if done:
break
episode_rewards.append(episode_reward)
if (episode + 1) % 10 == 0:
print(
f"{'DDQN' if double_dqn else 'DQN'} - Episode {episode+1}/{num_episodes}, "
f"Reward: {episode_reward:.1f}, Epsilon: {epsilon:.3f}"
)
env.close()
return episode_rewards
# Run DQN vs Double DQN
if __name__ == "__main__":
random.seed(0)
np.random.seed(0)
torch.manual_seed(0)
print("Training standard DQN...")
rewards_dqn = train_agent(double_dqn=False)
print("\nTraining Double DQN...")
rewards_ddqn = train_agent(double_dqn=True)
# Simple smoothing
def smooth(x, w=10):
if len(x) < w:
return x
return np.convolve(x, np.ones(w) / w, mode="valid")
plt.figure(figsize=(8, 5))
plt.plot(smooth(rewards_dqn), label="DQN")
plt.plot(smooth(rewards_ddqn), label="Double DQN")
plt.xlabel("Episode")
plt.ylabel("Reward")
plt.title("CartPole-v1: DQN vs Double DQN")
plt.legend()
plt.tight_layout()
plt.show()This is also done for the LunarLander environment,
# Q-network: approximates Q(s,a)
class QNetwork(nn.Module):
def __init__(self, state_dim, action_dim):
super().__init__()
# Simple 2-hidden-layer MLP
self.net = nn.Sequential(
nn.Linear(state_dim, 256),
nn.ReLU(),
nn.Linear(256, 256),
nn.ReLU(),
nn.Linear(256, action_dim),
)
def forward(self, x):
return self.net(x)
# Experience Replay Buffer
# Stores transitions to break correlation
class ReplayBuffer:
def __init__(self, capacity):
self.buffer = collections.deque(maxlen=capacity)
def push(self, state, action, reward, next_state, done):
# Add latest transition
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size):
# Random mini-batch sample
batch = random.sample(self.buffer, batch_size)
states, actions, rewards, next_states, dones = zip(*batch)
return (
np.array(states, dtype=np.float32),
np.array(actions, dtype=np.int64),
np.array(rewards, dtype=np.float32),
np.array(next_states, dtype=np.float32),
np.array(dones, dtype=np.float32),
)
def __len__(self):
return len(self.buffer)
# Training Loop: DQN or Double DQN
def train_lunar_lander(
env_id="LunarLander-v3",
num_episodes=800,
gamma=0.99,
batch_size=64,
buffer_capacity=100_000,
lr=1e-3,
target_update_freq=1000,
min_buffer_size=5000,
double_dqn=False,
):
"""
Train an agent using either:
- standard DQN (max over target network)
- Double DQN (DDQN) (action selection online, value evaluation target)
"""
env = gym.make(env_id)
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device} | Mode: {'Double DQN' if double_dqn else 'DQN'}")
online_net = QNetwork(state_dim, action_dim).to(device)
target_net = QNetwork(state_dim, action_dim).to(device)
target_net.load_state_dict(online_net.state_dict())
target_net.eval()
optimizer = optim.Adam(online_net.parameters(), lr=lr)
replay_buffer = ReplayBuffer(buffer_capacity)
# Epsilon-greedy exploration schedule
epsilon_start = 1.0
epsilon_final = 0.05
epsilon_decay = 200_000 # decay across steps, not episodes
global_step = 0
episode_rewards = []
max_steps_per_episode = 1000
# Main training loop
for episode in range(num_episodes):
state, info = env.reset()
episode_reward = 0.0
for t in range(max_steps_per_episode):
global_step += 1
# Compute epsilon as exponential decay
epsilon = epsilon_final + (epsilon_start - epsilon_final) * np.exp(
-1.0 * global_step / epsilon_decay
)
# Exploration vs exploitation choice
if random.random() < epsilon:
action = env.action_space.sample()
else:
with torch.no_grad():
s = torch.tensor(state, dtype=torch.float32, device=device).unsqueeze(0)
q_values = online_net(s) # Q(s,a) estimates
action = int(torch.argmax(q_values, dim=1).item())
# Step the environment
next_state, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated
# Store transition in replay buffer
replay_buffer.push(state, action, reward, next_state, done)
state = next_state
episode_reward += reward
# Learning step (if enough data collected)
if len(replay_buffer) >= min_buffer_size:
(
states,
actions,
rewards,
next_states,
dones,
) = replay_buffer.sample(batch_size)
# Convert arrays → tensors
states = torch.tensor(states, dtype=torch.float32, device=device)
actions = torch.tensor(actions, dtype=torch.long, device=device)
rewards = torch.tensor(rewards, dtype=torch.float32, device=device)
next_states = torch.tensor(next_states, dtype=torch.float32, device=device)
dones = torch.tensor(dones, dtype=torch.float32, device=device)
# Q-values for chosen actions: Q(s,a)
q_values = online_net(states)
q_sa = q_values.gather(1, actions.unsqueeze(1)).squeeze(1)
# Compute targets
with torch.no_grad():
if double_dqn:
# Double DQN target
# Step 1 - choose best action using ONLINE net (argmax)
next_q_online = online_net(next_states)
next_actions = torch.argmax(next_q_online, dim=1, keepdim=True)
# Step 2 - evaluate that action using TARGET net
next_q_target = target_net(next_states)
next_q = next_q_target.gather(1, next_actions).squeeze(1)
else:
# Standard DQN target
# Directly take max Q-value from target net
next_q_target = target_net(next_states)
next_q, _ = next_q_target.max(dim=1)
# Bellman target y = r + γ * Q( s', a' )
targets = rewards + gamma * next_q * (1.0 - dones)
# Loss = MSE between current estimate and target value
loss = nn.MSELoss()(q_sa, targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Periodically sync target net with online net
if global_step % target_update_freq == 0:
target_net.load_state_dict(online_net.state_dict())
if done:
break
episode_rewards.append(episode_reward)
# Logging every 10 episodes
if (episode + 1) % 10 == 0:
print(
f"{'DDQN' if double_dqn else 'DQN'} | "
f"Episode {episode+1}/{num_episodes} | "
f"Reward: {episode_reward:.1f} | Epsilon: {epsilon:.3f}"
)
env.close()
return episode_rewards, online_net
# Watch trained agent (optional)
def watch_agent(env_id, policy_net, episodes=3):
env = gym.make(env_id, render_mode="human")
device = next(policy_net.parameters()).device
for ep in range(episodes):
state, info = env.reset()
done = False
total_reward = 0.0
while not done:
with torch.no_grad():
s = torch.tensor(state, dtype=torch.float32, device=device).unsqueeze(0)
q_values = policy_net(s)
action = int(torch.argmax(q_values, dim=1).item())
state, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated
total_reward += reward
print(f"Episode {ep+1}: total reward = {total_reward:.1f}")
env.close()
# Main: Train & Compare
if __name__ == "__main__":
random.seed(0)
np.random.seed(0)
torch.manual_seed(0)
# Train standard DQN
print("Training DQN...")
rewards_dqn, net_dqn = train_lunar_lander("LunarLander-v3", num_episodes=800, double_dqn=False)
# Train Double DQN
print("\nTraining Double DQN...")
rewards_ddqn, net_ddqn = train_lunar_lander("LunarLander-v3", num_episodes=800, double_dqn=True)
# Helper to smooth curves
def smooth(x, w=20):
if len(x) < w:
return x
return np.convolve(x, np.ones(w) / w, mode="valid")
# Plot learning curves
plt.figure(figsize=(8, 5))
plt.plot(smooth(rewards_dqn), label="DQN")
plt.plot(smooth(rewards_ddqn), label="Double DQN")
plt.xlabel("Episode")
plt.ylabel("Reward")
plt.title("LunarLander-v3: DQN vs Double DQN Learning Curve")
plt.legend()
plt.tight_layout()
plt.show()Results and Visualisation
The Cartpole-v1 was run in DQN and DDQN for both 500 and 5000 episodes.
In the graph above, it is clear that both agents learn to solve the task as rewards rise sharply between 1000–2000 episodes. Both frequently reach the maximum reward of 500, meaning they both balance the pole for the full episode.
Get Anika Ip’s stories in your inbox
Join Medium for free to get updates from this writer.
Double DQN learns faster, shown by the steeper and earlier initial ramp of the orange line, but both performances become volatile once the problem is solved — oscillating between rewards of 200–500 after 2000 episodes. This is because the problem is technically already solved, but both agents are still attempting to explore, which destabilises the rewards. It can also be seen that the DQN rewards shows several sharp drops to 50–100 near episodes 3500–5000, whereas DDQN stays consistently high because of its reduced overestimation and more stable Q-values.
Since LunarLander is a more complicated problem, there is no early rapid climb like the CartPole results. Both algorithms start around –200 reward, often crashing, and meaningful improvements are only visible after 200–300 episodes. However, it can be seen that DDQN is consistently better, Orange curve rises higher than blue after around 400 episodes, stabilising around –80 to –90 while DQN often dips back to ~ –120 or worse, fluctuating wildly even after long training. This matches the theory discussed earlier: DQN overestimates high-risk, fuel-burning actions, leading to crashes; Double DQN makes more conservative estimates, leading to fewer catastrophic failures. Neither fully solve LunarLander within 800 episodes, but trends clearly show that DDQN is outperforming DQN.
Evaluation
On CartPole, DQN and Double DQN can both solve the task, and the performance difference is often modest.
On LunarLander, standard DQN tends to overestimate Q-values, leading to issues such as too aggressive thrust usage and crashes from overconfident policies, which ultimately mean unstable returns.
Double DQN learns safer, more conservative policies, usually converging more smoothly and achieving higher average reward.
Bioengineering Contextualisation
Reinforcement learning is increasingly used in bioengineering applications that involve sequential decision-making under uncertainty — especially where patient safety is critical. Double DQN is particularly useful in such contexts because it avoids risky overestimation of high-reward actions.
Example
One example could be an RL agent adjusting drug dosage in response to patient-specific biomarker levels. The state may include indicators such as inflammation markers, drug concentration, therapeutic response, and estimated toxicity risk. The available actions could be: increase dose, maintain dose, or decrease dose.
The reward could be designed as follows: positive for moving biomarkers towards therapeutic targets, negative for side effects or exceeding safe concentration thresholds.
Here, overestimation is dangerous: if the agent overestimates the value of a higher dose, it may repeatedly choose aggressive dosing strategies that risk toxicity. Double DQN’s reduced overestimation leads to safer, more conservative value estimates and therefore more reliable clinical decision policies.
Relevance
This approach aligns with emerging research in adaptive therapeutics, closed-loop pharmacokinetics, and intelligent drug delivery systems — key areas where safe and stable reinforcement learning algorithms are essential.
Summary
Double DQN is a simple yet powerful extension of DQN that addresses its main weakness — overestimation bias. By separating action selection (online network) from action evaluation (target network), Double DQN produces more realistic target values, which leads to improved learning stability and more reliable policies.
In this tutorial, we outlined: the motivation behind Double DQN, the key mathematical modification it introduces, a concise implementation, experimental results illustrating reduced overestimation, and a bioengineering application where stability and safety are critical.
The algorithm’s elegance lies in its minimal modification to DQN while offering significant improvements. Double DQN remains one of the most practical and widely used value-based methods in reinforcement learning today, especially in domains where safety and reliability are essential.
Achknowledgements
I acknowledge the use of ChatGPT 5 (OpenAI, https://chat.openai.com/) to generate an outline for background study and code checking. I confirm that no content generated by AI has been presented as my own work.
References
[1] Sutton, R. S. and Barto, A. G. “Reinforcement Learning: An Introduction”. In: The MIT Press (2014)
[2] ScienceDirect. “Deep Q-Networks”. Accessed via link: https://www.sciencedirect.com/topics/computer-science/deep-q-network
[3] ApX Courses. “Double DQN (DDQN)”. Accessed via link: https://apxml.com/courses/intermediate-reinforcement-learning/chapter-3-dqn-improvements-variants/double-dqn-ddqn
[4] ApX Courses. “The DQN Algorithm Architecture”. Accessed via link: https://apxml.com/courses/intermediate-reinforcement-learning/chapter-2-deep-q-networks-dqn
[5] ApX Courses. “The Overestimation Problem in Q-Learning”. Accessed via link: https://apxml.com/courses/intermediate-reinforcement-learning/chapter-3-dqn-improvements-variants/q-learning-overestimation
