When I first trained AlphaBoop (an AlphaZero model of Boop), I did notice a problem.
In the later stages of training, the AI would become much slower. Granted, this is to be expected because as the AI grew stronger, the number of turns in their self-played games grew.
But, the increased training time was still abnormal. When calculating ELO, different iterations of the AI had to fight each other, and on very rare occasions, they appeared to be stuck.
There are instances when I pit two AI agents against each other, and they get trapped in an infinite loop.
In order to study the effects of infinite loops, I retrained AlphaZero from scratch and asserted a new rule:
For instance, you can have a completely blank board where each player has 8 Kittens. That is simply the start of the game.
However, you can also have a completely blank board where each player has 3 Cats and 5 Kittens. Although the board is identical, I am treating this as a different "board state".
Below is a graph that illustrates the rate of draws over the iterations of AlphaBoop's training, as you can see, this explains why my original training was slowing down because once I factored in this rule, the AI no longer gets struck anymore at the advanced stages of learning.
In the beginning, the rate of draws in the early iterations was simply 0% of the time. More accurately, in over 3000 self-played games, none of them resulted in a draw.
Then, gradually, you see a somewhat linear trend of growth as the rate of draws grew over time as the AI got stronger. The last recorded iteration marks the highest rate of draws which is 0.87% of the games in that iteration were drawn.
In addition, by asserting this new rule on "draws", the ELO spiked higher than my original training by a significant amount right at the starting iterations.
I suspect the reason is because the infinite loops dilute the exploration of the AI. It would be like exploring a forest, but you occasionally walk circles around a tree like an idiot - wasting your time and energy. By defining a hard stopping point for repeated board states, the AI likely gets to more efficiently conduct its search.
Interestingly, some of the moves that this AI (with the draw rule established) thinks are good are not the same as the original implementation.
For example, in response to C4:
This AI thinks there are two good moves: B2 and D3. D3 is very standard, yet B2 seems unusual. Originally, the AI thought D3 was THE best move.
Interestingly, when I trained Gekitai (the "Young Player Variant" of Boop), playing B2 was considered the best opening response to C4 according to the AI.
When I crank up the Number of Simulations to 16,000, B2 and D3 are rated as comparable. When I crank it up to 64,000, suddenly B2 became favored more.
So... if any of you are brave enough to experiment B2 as a response to C4, be my guest. Maybe my AI is just stupid.
In response to B2, the AI says there are four moves it would consider: D4 (highest rated), A3, B4, and C3. The AI would play D4 here:
In the later stages of training, the AI would become much slower. Granted, this is to be expected because as the AI grew stronger, the number of turns in their self-played games grew.
But, the increased training time was still abnormal. When calculating ELO, different iterations of the AI had to fight each other, and on very rare occasions, they appeared to be stuck.
There are instances when I pit two AI agents against each other, and they get trapped in an infinite loop.
In order to study the effects of infinite loops, I retrained AlphaZero from scratch and asserted a new rule:
To clarify, my definition of a board state isn't merely what you see on a 6x6 board. It includes the number of kittens and cats that each player has in their pool.If a board state occurs three times, the game is a draw.
For instance, you can have a completely blank board where each player has 8 Kittens. That is simply the start of the game.
However, you can also have a completely blank board where each player has 3 Cats and 5 Kittens. Although the board is identical, I am treating this as a different "board state".
Below is a graph that illustrates the rate of draws over the iterations of AlphaBoop's training, as you can see, this explains why my original training was slowing down because once I factored in this rule, the AI no longer gets struck anymore at the advanced stages of learning.
In the beginning, the rate of draws in the early iterations was simply 0% of the time. More accurately, in over 3000 self-played games, none of them resulted in a draw.
Then, gradually, you see a somewhat linear trend of growth as the rate of draws grew over time as the AI got stronger. The last recorded iteration marks the highest rate of draws which is 0.87% of the games in that iteration were drawn.
In addition, by asserting this new rule on "draws", the ELO spiked higher than my original training by a significant amount right at the starting iterations.
I suspect the reason is because the infinite loops dilute the exploration of the AI. It would be like exploring a forest, but you occasionally walk circles around a tree like an idiot - wasting your time and energy. By defining a hard stopping point for repeated board states, the AI likely gets to more efficiently conduct its search.
Interestingly, some of the moves that this AI (with the draw rule established) thinks are good are not the same as the original implementation.
For example, in response to C4:
This AI thinks there are two good moves: B2 and D3. D3 is very standard, yet B2 seems unusual. Originally, the AI thought D3 was THE best move.
Interestingly, when I trained Gekitai (the "Young Player Variant" of Boop), playing B2 was considered the best opening response to C4 according to the AI.
When I crank up the Number of Simulations to 16,000, B2 and D3 are rated as comparable. When I crank it up to 64,000, suddenly B2 became favored more.
So... if any of you are brave enough to experiment B2 as a response to C4, be my guest. Maybe my AI is just stupid.
In response to B2, the AI says there are four moves it would consider: D4 (highest rated), A3, B4, and C3. The AI would play D4 here:
