Playing repeated games with Large Language Models

The paper follows a clear and logical structure, moving from a broad analysis of LLM performance across different game families to a detailed examination of specific games, interventions (SCoT), and finally human-LLM interactions. The narrative is easy to follow and build incrementally.

Strengths: Detailed description of the game setup, LLMs used, and prompt construction., Use of multiple LLMs and comparison to simple strategies., Inclusion of multiple robustness checks to verify behavioural signatures., Rigorous design and statistical analysis of the human participant study., Clear definition of performance metrics.
Weaknesses: Parametric test assumptions (normal distribution) were not formally tested, although standard practice.

The claims are well-supported by quantitative data presented in tables and figures, covering extensive simulations of LLM-LLM interactions across many games and a human study with a reasonable sample size. Robustness checks provide further support for the stability of the observed behaviours.

The approach of applying behavioural game theory on a large scale to analyze LLMs' social behaviour is highly original. The identification of specific behavioural signatures (unforgivingness in PD, failure to alternate in BoS) and the introduction of social chain-of-thought prompting as an intervention are novel contributions.

The research addresses a timely and important topic (understanding LLM behaviour in social contexts) with direct implications for human-AI interaction, AI alignment, and the broader field of machine behaviour. The findings provide concrete steps for improving LLM social capabilities.

Strengths: Formal and precise academic language., Key concepts and game types are clearly introduced., Methodology and results are described in sufficient detail for the target audience.
Areas for Improvement: None