Using Multi-Agent Reinforcement Learning to play OpenSpiel's Connect 4 with Ray's RLlib

Introduction

Connect 4, a timeless two-player game, has intrigued both casual enthusiasts and AI researchers alike. This post walks through training a Connect 4 agent with PPO on Ray RLlib and OpenSpiel, using league-based self-play — the agent's opponent is a frozen copy of an earlier version of itself, replaced with a stronger one each time it wins too easily. Self-play enables agents to improve iteratively by competing against themselves, eliminating the need for external opponents.

What is OpenSpiel?

OpenSpiel, developed by DeepMind, is a comprehensive library of games and algorithms for RL research. It supports a wide range of games, including board games like Chess, Go, and Connect 4. With built-in tools for self-play and algorithmic experimentation, OpenSpiel is an ideal choice for developing AI agents in adversarial game environments.

Why Self-Play?

Self-play is a cornerstone of RL in competitive environments. By playing against itself, an agent continuously refines its strategies by learning from victories and defeats. Landmark achievements such as AlphaGo and AlphaZero underscore the efficacy of this approach. In Connect 4, self-play allows the agent to master complex strategies through iterative improvement.

Formalizing Self-Play

Self-play in reinforcement learning can be formalized using game theory and Markov Decision Processes (MDPs). Let’s break it down step by step:

1. Game Setup: A two-player zero-sum game like Connect 4 is represented as a tuple :
   • : Set of states, representing all possible configurations of the board.
   • : Set of actions, corresponding to the legal moves in the game.
   • : State transition probabilities when action  is taken in state .
   • : Reward function, defining the payoff for taking action  in state .
   • : Discount factor, indicating the importance of future rewards.
2. Self-Play Dynamics: During training, each agent alternates between being the active player and the opponent. This dynamic ensures that the agents optimize their policies, , against a continuously improving adversary.
3. Optimization Objective: The goal is to find a Nash equilibrium where neither agent can unilaterally improve its performance. Formally, this is achieved by minimizing regret:Here,  is the optimal value of state , and  is the value of taking action  in state  under the agent’s current policy.
4. Learning from Outcomes: At the end of each episode, both agents update their policies based on the observed rewards and state transitions. Techniques like Q-learning or policy gradients can be employed for this purpose.

AlphaZero: The Inspiration

AlphaZero, developed by DeepMind, revolutionized the use of self-play and deep reinforcement learning in adversarial games. It builds upon the Monte Carlo Tree Search (MCTS) framework combined with neural networks to approximate policies and value functions. Here’s how AlphaZero’s core concepts relate to Connect 4 and Ray RLlib:

1. Policy and Value Networks: AlphaZero uses a deep neural network to predict the policy (the probability of each action) and the value (expected outcome) for any given game state.
2. Monte Carlo Tree Search (MCTS): During gameplay, AlphaZero employs MCTS to explore potential future states, balancing exploration and exploitation. The neural network guides the search, reducing computational costs compared to traditional methods.
3. Self-Play: By playing against itself, AlphaZero generates high-quality data to train its neural networks. Over time, the policy improves, enabling the agent to outperform traditional approaches.
4. Generalization Across Games: AlphaZero demonstrated that the same algorithm could achieve superhuman performance in Chess, Shogi, and Go, showcasing its adaptability.

How RLlib Runs League-Based Self-Play

RLlib does not ship an AlphaZero implementation, and this experiment does not build one. There is no tree search here. What it uses instead is league-based self-play with PPO, which keeps AlphaZero's central idea — the opponent improves as you do — while replacing MCTS with a policy-gradient learner.

The setup has three moving parts:

1. Two policies to start. A trainable policy called main and a fixed random policy. Only main is ever trained.
2. A mapping function that alternates seats. Connect 4 has a first-player advantage, so the agent has to be good from either seat. Hashing the episode id decides which side main takes, which keeps the assignment even across episodes.
3. A callback that grows a league. SelfPlayCallback watches the rolling 100-episode win rate. When it crosses the threshold, the current weights are frozen as a new snapshot — main_v0, main_v1, and so on — and added to the opponent pool. The agent stops facing a random opponent and starts facing its own past selves.

That last piece is what makes the win rate a moving target: every time main gets comfortable, the bar moves.

The Configuration

The experiment is driven from a small set of values. The two that matter most are the win-rate threshold that triggers a snapshot and the league size that ends the run.

@dataclass
class Args:
    env: str = "connect_four"
    algo: str = "PPO"

    # League self-play
    win_rate_threshold: float = 0.95
    min_league_size: int = 3

    # Stopping criteria
    stop_timesteps: int = 2_000_000
    stop_iters: int = 100

    # Compute
    num_env_runners: int = 2
    num_learners: int = 1
    num_gpus_per_learner: int = 1
    framework: str = "torch"

Training stops on whichever comes first: two million environment steps, one hundred iterations, or a league of three. In practice the league fills long before either of the other two.

Building the RLlib Config

OpenSpiel's Connect 4 is registered through RLlib's OpenSpielEnv wrapper, so the game itself needs no custom environment code.

register_env(
    "open_spiel_env",
    lambda _: OpenSpielEnv(pyspiel.load_game(args.env)),
)


def agent_to_module_mapping_fn(agent_id, episode, **kwargs):
    # Hash the episode id so "main" plays first and second about equally
    # often, and always against the opponent rather than itself.
    return "main" if hash(episode.id_) % 2 == agent_id else "random"


config = (
    get_trainable_cls(args.algo)
    .get_default_config()
    .environment("open_spiel_env")
    .callbacks(
        functools.partial(
            SelfPlayCallback,
            win_rate_threshold=args.win_rate_threshold,
        )
    )
    .env_runners(
        num_env_runners=(args.num_env_runners or 2),
        num_envs_per_env_runner=1,
    )
    .multi_agent(
        policies={"main", "random"},
        policy_mapping_fn=agent_to_module_mapping_fn,
        policies_to_train=["main"],
    )
    .rl_module(
        model_config=DefaultModelConfig(fcnet_hiddens=[512, 512]),
        rl_module_spec=MultiRLModuleSpec(
            rl_module_specs={
                "main": RLModuleSpec(),
                "random": RLModuleSpec(module_class=RandomRLModule),
            }
        ),
    )
)

if args.algo == "PPO":
    config.training(num_epochs=20)

stop = {
    NUM_ENV_STEPS_SAMPLED_LIFETIME: args.stop_timesteps,
    TRAINING_ITERATION: args.stop_iters,
    "league_size": args.min_league_size,
}

The policy network is deliberately plain: two fully connected layers of 512 units. Connect 4's board is small enough that the interesting part of this experiment is the opponent schedule, not the architecture.

Training

With the config assembled, the run itself is one line.

results = run_rllib_example_script_experiment(config, args, stop=stop)

Results

On a Google Colab L4:

Two things are worth reading carefully here. The first is how little compute this took: 172,000 environment steps in about four and a half minutes. Connect 4 has roughly 4.5 trillion legal positions, so nothing here is memorising the game — the network is learning a positional evaluation that generalises.

The second is what the 0.97 actually measures. It is the rolling 100-episode win rate that SelfPlayCallback sees against the current league opponent, which is the signal that triggers a new snapshot. It is not a fixed benchmark, and it is not a claim of strength against a strong opponent. Because the opponent pool keeps changing, the number says the agent kept clearing the bar it was set, not that it plays Connect 4 well in absolute terms.

For fixed baselines the notebook evaluates the trained policy against a uniform-random policy and against every league snapshot after training, and writes win, loss and draw rates with 95% confidence intervals to results.json, alongside learning curves, an opening-move distribution and MP4 replays of the agent playing. Those artifacts are written per run rather than committed, so reproduce them by running the notebook.

One diagnostic in there is worth calling out. Connect 4 is solved: the first player wins with perfect play by opening in the centre column. The notebook plots the trained agent's opening move over 200 deterministic rollouts, which gives a direct read on whether self-play rediscovered that on its own.

Key Concepts in Self-Play

1. Exploration vs. Exploitation: Balancing exploration of new moves with exploitation of successful strategies is crucial. Techniques like ε-greedy help manage this trade-off.
2. Learning from Defeats: Self-play fosters resilience by enabling agents to learn from losing games, not just victories.
3. Opponent Adaptation: Agents iteratively refine strategies by exploiting weaknesses in their prior behaviors, generating robust tactics.

What’s Next?

The obvious extensions from here:

• Raise min_league_size so the agent faces a deeper bench of past selves rather than stopping at three.
• Add MCTS on top of the learned policy, which is the piece that separates this from AlphaZero proper.
• Compare against a solver to turn "wins against its own history" into an absolute strength measure.
• Swap PPO for another learner and hold the league schedule fixed, to see how much of the result is the algorithm and how much is the opponent curriculum.

Conclusion

League-based self-play gets a PPO agent playing credible Connect 4 in about four and a half minutes of training on a single L4, without a tree search, a hand-written opponent, or a single labelled game. The opponent schedule is doing most of the work: freezing a snapshot every time the agent clears a 95% win rate turns a fixed problem into a moving one, and the agent has to keep improving to stay ahead of itself.

What the run does not establish is absolute strength. The headline win rate is measured against the league, so it says the agent kept beating its own past selves — a self-referential benchmark. Connect 4 is a solved game, so the honest next step is to measure against a solver, or at least against the fixed random baseline the notebook already evaluates, and see how much of that 0.97 survives contact with an opponent that is not a version of itself.

Additional Learning Materials

• Self-Play: a classic technique to train competitive agents in adversarial games
• OpenSpiel Documentation
• Ray Documentation
• AlphaZero: Shedding new light on chess, shogi, and Go

Code Repository & Models

• Github Repository
• Google Colab Notebook