Analyzing KataGo: A Comparative Evaluation Against Perfect Play in the Game of Go

Abstract

Research on board games focuses on playing at a superhuman level or finding exact solutions. Recently, Artificial Intelligence (AI) has become really good at playing complex games such as Go. Comparing AI systems to perfect play helps us understand how advanced AI has become. This research explores the performance of KataGo, an AlphaZero-like program, in the game of Go. Our study investigates how different neural networks and search strategies impact KataGo’s decision-making abilities when compared against perfect play. In our research, we develop a larger Go endgame dataset labelled with perfect solutions, and examine KataGo’s strengths and weaknesses through experiments and analysis. We observe the effectiveness of strong policies in improving move selection, the benefits and demerits of MCTS search enhancements, and the challenges KataGo faces in competing against an exact solver. We further analyse move choices by showing the changes of average action values, lower confidence bound (lcb), winrate, and number of visited node according to MCTS search in KataGo. KataGo has a 90.8% success rate while playing matches against an exact solver in the perfect game dataset.