Deep RL: DQN, PPO, SAC, and Multi-Agent Algorithms
Deep reinforcement learning combines the representational power of neural networks with reinforcement learning algorithms, enabling agents to solve complex decision-making problems that were previously intractable. Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC) are the three most influential algorithms in modern deep RL, each representing a different approach to the core challenge of learning effective policies from high-dimensional observations.
The field has progressed rapidly since DQN achieved human-level performance on Atari games in 2015. AlphaGo used deep RL to defeat the world champion in Go — a game once thought decades away from machine mastery. PPO has become the default algorithm for continuous control tasks in robotics and simulation. SAC adds entropy maximization for better exploration, and multi-agent RL extends these ideas to systems with multiple interacting agents.
Deep Q-Networks
DQN (Mnih et al., 2015) was the first algorithm to successfully combine neural networks with Q-learning, using raw pixel inputs from Atari games. Its key innovations addressed the fundamental instability of combining non-linear function approximation with off-policy learning.
Experience Replay Implementation
Experience replay stores transitions in a buffer and samples randomly to decorrelate consecutive experiences:
import random
import numpy as np
from collections import deque
class ReplayBuffer:
def __init__(self, capacity: int = 100000):
self.buffer = deque(maxlen=capacity)
def push(self, state, action, reward, next_state, done):
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size: int):
batch = random.sample(self.buffer, batch_size)
states, actions, rewards, next_states, dones = map(np.stack, zip(*batch))
return states, actions, rewards, next_states, dones
def __len__(self):
return len(self.buffer)Target Network and Training Loop
A separate target network provides stable Q-value targets. The target network parameters are copied from the online network at regular intervals:
import torch
import torch.nn as nn
import torch.nn.functional as F
class DQNAgent:
def __init__(self, state_dim, action_dim, hidden_dim=256):
self.q_network = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, action_dim)
)
self.target_network = nn.Sequential(*[copy.deepcopy(layer) for layer in self.q_network])
self.target_network.eval()
self.optimizer = torch.optim.Adam(self.q_network.parameters(), lr=3e-4)
self.gamma = 0.99
self.buffer = ReplayBuffer(capacity=100000)
self.batch_size = 64
self.target_update_freq = 1000
self.steps = 0
def update(self):
if len(self.buffer) < self.batch_size:
return
states, actions, rewards, next_states, dones = self.buffer.sample(self.batch_size)
states = torch.FloatTensor(states)
actions = torch.LongTensor(actions)
rewards = torch.FloatTensor(rewards)
next_states = torch.FloatTensor(next_states)
dones = torch.FloatTensor(dones)
# Current Q values
current_q = self.q_network(states).gather(1, actions.unsqueeze(1)).squeeze()
# Target Q values using frozen target network
with torch.no_grad():
max_next_q = self.target_network(next_states).max(1)[0]
target_q = rewards + self.gamma * max_next_q * (1 - dones)
# Loss and update
loss = F.mse_loss(current_q, target_q)
self.optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self.q_network.parameters(), 1.0)
self.optimizer.step()
# Periodic target network update
self.steps += 1
if self.steps % self.target_update_freq == 0:
self.target_network.load_state_dict(self.q_network.state_dict())Prioritized experience replay (Schaul et al., 2016) improves on uniform sampling by sampling more frequently from transitions with larger TD errors — the ones the agent has most to learn from.
Proximal Policy Optimization
PPO (Schulman et al., 2017) has become the default algorithm for continuous control due to its stability, simplicity, and strong performance. It introduces a clipped surrogate objective that prevents policy updates from being too large:
def compute_ppo_loss(actor, critic, states, actions, old_log_probs, advantages, returns, epsilon=0.2):
# New log probabilities under current policy
action_probs = actor(states)
dist = torch.distributions.Categorical(action_probs)
log_probs = dist.log_prob(actions)
# Probability ratio
ratio = torch.exp(log_probs - old_log_probs)
# Clipped surrogate objective
surr1 = ratio * advantages
surr2 = torch.clamp(ratio, 1.0 - epsilon, 1.0 + epsilon) * advantages
actor_loss = -torch.min(surr1, surr2).mean()
# Value function loss
values = critic(states).squeeze()
critic_loss = F.mse_loss(values, returns)
# Entropy bonus (optional, encourages exploration)
entropy = dist.entropy().mean()
entropy_coef = 0.01
total_loss = actor_loss + 0.5 * critic_loss - entropy_coef * entropy
return total_lossThe clipping mechanism restricts the probability ratio to [1-epsilon, 1+epsilon], meaning the new policy cannot deviate too far from the old policy in a single update. This prevents the destructive policy updates that plagued earlier policy gradient methods. PPO typically performs multiple epochs of mini-batch updates on the same trajectory data (typically 3-10 epochs), making it sample-efficient compared to pure on-policy methods.
PPO Training Loop
class PPOAgent:
def __init__(self, state_dim, action_dim):
self.actor = self._build_network(state_dim, action_dim)
self.critic = self._build_network(state_dim, 1)
self.optimizer = torch.optim.Adam(
list(self.actor.parameters()) + list(self.critic.parameters()),
lr=3e-4
)
self.gamma = 0.99
self.gae_lambda = 0.95
self.ppo_epochs = 10
self.clip_epsilon = 0.2
def update(self, trajectories):
states, actions, rewards, dones, values, log_probs = trajectories
# Compute advantages using Generalized Advantage Estimation
advantages = []
gae = 0
for t in reversed(range(len(rewards))):
if t == len(rewards) - 1:
next_value = 0 # Terminal state
else:
next_value = values[t + 1]
delta = rewards[t] + self.gamma * next_value * (1 - dones[t]) - values[t]
gae = delta + self.gamma * self.gae_lambda * (1 - dones[t]) * gae
advantages.insert(0, gae)
advantages = torch.FloatTensor(advantages)
returns = advantages + torch.FloatTensor(values)
advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
# Multiple PPO epochs
for _ in range(self.ppo_epochs):
loss = compute_ppo_loss(
self.actor, self.critic,
states, actions, log_probs,
advantages, returns, self.clip_epsilon
)
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()Soft Actor-Critic
SAC (Haarnoja et al., 2018) adds entropy maximization to the objective function. The agent learns to maximize both expected return and policy entropy, encouraging exploration and preventing premature convergence to narrow policies:
def sac_update(actor, qf1, qf2, value, target_value, batch, alpha, gamma=0.99):
states, actions, rewards, next_states, dones = batch
# Compute target Q value
with torch.no_grad():
next_actions, next_log_probs = actor.sample(next_states)
next_q = torch.min(
qf1(next_states, next_actions),
qf2(next_states, next_actions)
)
target_q = rewards + gamma * (1 - dones) * (next_q - alpha * next_log_probs)
# Q-function losses (clipped double Q-learning)
qf1_loss = F.mse_loss(qf1(states, actions), target_q)
qf2_loss = F.mse_loss(qf2(states, actions), target_q)
# Policy loss with entropy
new_actions, log_probs = actor.sample(states)
q_values = torch.min(
qf1(states, new_actions),
qf2(states, new_actions)
)
actor_loss = (alpha * log_probs - q_values).mean()
# Temperature parameter alpha (auto-tuned)
alpha_loss = -(alpha.log() * (log_probs.detach() + target_entropy).mean())SAC uses two Q-functions and takes the minimum to reduce overestimation bias — a problem where Q-learning systematically overestimates action values. The entropy coefficient alpha can be automatically tuned to a target entropy value.
Multi-Agent Reinforcement Learning
In multi-agent environments, each agent’s policy must account for the non-stationarity caused by other agents learning and changing their behavior simultaneously. MADDPG (Lowe et al., 2017) uses a centralized critic that observes all agents’ actions, with decentralized actors that only use local observations:
class MADDPG:
def __init__(self, n_agents, obs_dims, action_dims):
self.agents = []
for i in range(n_agents):
agent = DDPGAgent(obs_dims[i], action_dims[i])
self.agents.append(agent)
def update(self, experiences, agent_id):
# Centralized critic sees all observations and actions
all_obs = torch.cat([exp['obs'] for exp in experiences], dim=-1)
all_actions = torch.cat([exp['action'] for exp in experiences], dim=-1)
# Update critic with global information
target_q = self.compute_target_q(experiences, agent_id)
current_q = self.agents[agent_id].critic(all_obs, all_actions)
critic_loss = F.mse_loss(current_q, target_q)
# Actor uses only local observation
local_obs = experiences[agent_id]['obs']
actor_loss = -self.agents[agent_id].critic(
all_obs,
self.combine_actions(experiences, agent_id)
).mean()Hyperparameter Sensitivity
Deep RL algorithms are notoriously sensitive to hyperparameters. The learning rate, discount factor, entropy coefficient, network architecture, and buffer size all interact in complex ways. Key guidelines:
- Learning rate: 3e-4 is a good default for Adam across DQN, PPO, and SAC
- Discount factor (gamma): 0.99 for long-horizon tasks, 0.9-0.95 for shorter episodes
- Network width: 256-512 units per hidden layer for most continuous control tasks
- Replay buffer size: 1e5 for simple tasks, 1e6 for complex environments
- Target network update frequency: every 1000 steps for DQN
Population-based training and Bayesian optimization are more effective than manual tuning for scaling RL hyperparameter search.
FAQ
Which deep RL algorithm should I start with? PPO is the most practical starting point — it works well across both discrete and continuous action spaces, requires minimal hyperparameter tuning, and is relatively stable. DQN is limited to discrete actions. SAC generally outperforms PPO on continuous control but is more complex.
How do I handle continuous action spaces? DQN cannot handle continuous actions directly. Use PPO or SAC for continuous control. Both output parameters of a probability distribution (typically Gaussian) from which actions are sampled. For high-dimensional continuous actions, SAC with automatic entropy tuning is often the best choice.
What is the sample efficiency of these algorithms? From most to least sample efficient: SAC > DQN > PPO. SAC and DQN are off-policy (reuse past experiences). PPO is on-policy (uses each experience once). However, PPO’s multi-epoch updates within each trajectory partially offset the sample efficiency gap.
Why does RL training oscillate or diverge? Common causes: learning rate too high, reward scaling too large, insufficient exploration, or catastrophic forgetting where the agent loses previously learned skills. Use gradient clipping, reward normalization, and maintain multiple parallel environments to stabilize training.
Can I train deep RL on a laptop CPU? Simple environments like CartPole, MountainCar, and LunarLander train in minutes on CPU. Atari games benefit from GPU (hours per experiment). MuJoCo continuous control tasks need GPU for reasonable iteration speed.
Internal Links
- For foundational RL concepts and Q-learning, see Reinforcement Learning: Agents, Rewards, and Policy.
- For the neural network architectures underlying these algorithms, read Neural Networks: Perceptrons, Activation, and Backpropagation.
- For choosing the right framework to implement deep RL, see PyTorch vs TensorFlow: Choosing Your Framework.
Related Articles
AI & Machine Learning
Reinforcement Learning: Agents, Environments, and Rewards
Learn reinforcement learning including Markov decision processes, Q-learning, policy gradients, deep RL, and applications in robotics and games.
Machine Learning
Reinforcement Learning: Agents, Rewards, and Q-Learning
Learn reinforcement learning — MDPs, Q-learning, deep Q-networks, policy gradients, PPO, and practical RL applications in games, robotics, and decision systems.
AI & LLMs
AI and LLMs: Complete Guide to Large Language Models
Understand how large language models work — transformer architecture, training, tokenization, context windows, and practical applications. Based on the foundational 'Attention Is All You Need' paper (Vaswani et al., 2017).
AI & Machine Learning
Deep Learning: Neural Networks, Architectures, Training
Understand deep learning architectures including CNNs, RNNs, and transformers, plus training techniques like backpropagation and regularization.
AI & Machine Learning
Neural Networks: Architecture, Activations, and Training
Learn how neural networks work, from perceptrons and activation functions to backpropagation, gradient descent, and multi-layer architectures.