Design of Ataraxos
Ataraxos is a sophisticated AI framework developed for the strategic board game Stratego. It integrates self-play reinforcement learning processes facilitated by transformer networks. Below is an outline of its key components and functionalities:
1. Self-Play Reinforcement Learning Module
- Two Interdependent Processes:
- Set-Up Selection: Determines the starting positions for game pieces.
- Move Selection: Alternates turns to move pieces.
These processes operate separately but influence each other; the outcome of moves informs the set-ups, and vice versa.
2. Transformer Networks
- Network Types:
- Set-Up Network: A decoder-only transformer for learning set-ups. It employs Monte Carlo estimations and higher learning rates for effective training.
- Move Network: An encoder-only transformer for move selection which utilizes λ returns for estimating moves with an annealed learning rate for stability.
3. Self-Play Training Data Generation
- Game Simulation: Both set-ups and moves are sampled from their respective networks.
- Expected Returns Calculation: Estimations of expected values and advantages during training improve efficiency, reducing iterations significantly.
- Monte Carlo Returns for Set-ups: This method estimates performance based on game outcomes, enhancing learning accuracy.
4. Dynamically Damped Self-Play
- Policy Optimization: Ataraxos uses dynamically adjusted learning rates and regularization techniques to enhance training stability and control update sizes.
- Loss Regularization: Incorporation of entropy and Kullback-Leibler penalties mitigates overfitting and helps in policy shaping.
5. Belief Modelling
- Training a Belief Network: This network utilizes known game information to predict the hidden pieces of the opponent, enhancing strategizing capabilities.
- Transformer Architecture: Consists of an encoder to process known information and a decoder to make predictions about unknown pieces.
6. Test-Time Search Procedure
- Search Efficiency: Ataraxos incorporates an additional self-play update during search at test time to refine move predictions.
- Depth-Limited Rollouts: Evaluates potential moves by simulating a range of game states and estimating outcomes based on these samplings.
7. Implementation and Engineering Details
- Simulator Design: A GPU-accelerated simulator written in CUDA C++ focuses on performance and efficiency.
- Memory Management: A unified approach for simulating and tracking game states reduces memory overhead and enhances speed.
- Rules Enforcement: Custom algorithms ensure adherence to game-specific rules while maintaining efficiency during simulations.
8. Performance Evaluation
- Training Metrics: Ataraxos achieved significant performance metrics during its evaluation against top human players, showcasing its strategic prowess.
- Sample Efficiency: The AI demonstrated improvement over previous methodologies, achieving higher win rates with significantly lower computational resources.
9. Strengths and Play Style
- Aggressive Strategies: Showcases a unique approach to set-ups and gameplay, diverging from traditional human strategies.
- Positional Play: Exhibits remarkable abilities in long-term strategic thinking and adapting to complex game conditions.
10. Comparisons and Future Directions
- Cross-model Evaluation: Compared with DeepNash, Ataraxos showed superior performance and efficiency in training and gameplay.
- Exploration of New Architectures: Future enhancements could include integrating temporal attention or more advanced search methods to further improve performance.
11. Ethical Considerations
- Evaluation Studies: Oversight by the institutional review board ensures ethical compliance in testing human interactions with the AI.
This design outlines the comprehensive strategy and structure of Ataraxos, emphasizing its innovative approaches in reinforcement learning and AI architecture within the complex context of a competitive board game.
