Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, Cannada A. Lewis, Joel Veness, Tim Genewein

The paper demonstrates that Transformer models can effectively plan chess moves without relying on explicit search algorithms, which are central to state-of-the-art chess engines. Given the game’s complexity, chess presents a compelling challenge where neural networks cannot succeed through simple memorization of training data.

The dataset used in this study comprises 530 million board states with approximately 15 billion total legal next moves. Each move is labeled with its win probability, as calculated by Stockfish 16, the strongest publicly available chess engine. The authors released this dataset, derived from human games on lichess.org, under the name ChessBench. Board states are represented in FEN notation, which, while omitting the complete move sequence resulting in the given position, allows encoding any board state in a fixed-length context of about 80 tokens. They trained several decoder-only Transformer models, scaling up to 270 million parameters. Evaluation metrics were computed on a test set of around 10,000 chess puzzles from lichess.org, covering a wide range of Elo scores (a measure of puzzle difficulty).

The authors conducted an extensive ablation study with the following findings:

1. Prediction Targets: They explored three targets; the last one performed best:
   • Classifying the best legal move for a given position.
   • Predicting win probabilities for board states, indirectly yielding probabilities for legal next moves.
   • Directly predicting the win probability of each legal move by including candidate moves in the model’s context.
2. Label Smoothing and Binning: Binning probability predictions and applying label smoothing improved performance.
3. Model Depth: Increasing the network depth enhanced performance (up to a limit). The authors attribute that to improved iterative computations.

Remarkably, their biggest Transformer model performed almost on par with the AlphaZero chess engine (with half as many Monte Carlo tree search simulations as in the original paper) on the puzzle test set. The model was also tested on lichess.org, where it attained Elo scores of 2,300 against bots and 2,900 against humans. According to the authors, the discrepancy arises because Stockfish assigns a win probability of 100% to positions with a forced checkmate in n moves, while the Transformer lacks the ability to plan such strategies explicitly. This often resulted in accidental draws or even blunders resulting in ultimate losses. Human opponents tended to resign in losing positions, while bots played until checkmate.

A less encouraging result is the significant drop in playing strength when applying the model out-of-distribution to Fisher random chess (there the back row of the starting position is shuffled under the constraint that some basic rules of the game are observed, e.g., the ability to castle and opposite coloured bishops). In contrast, conventional chess engines are known to have little problems adapting to this variation.