How to spot players using AI? (works with other games)

[Kaepo]

I really like this game and its community of players on BGA. I'm sad to see strong players stop playing, but this loss is compensated by the constant arrival of new, talented players. So when someone I don't know arrives with a certain level, I naturally want to know more.

Almost two years ago, I saw the arrival of new, atypical players with strange and flamboyant styles, which certainly piqued my curiosity even more.

It turns out that this also coincided with the time when a deep learning-based AI was designed to play boop. Its creator discussed it on this very forum (https://boardgamearena.com/forum/viewtopic.php?t=34987), as well as on the BGG forum and on Reddit, from what I've seen.

This author did nothing wrong: he acted properly, asked BGA for permission to test his AI, as long as the opponents were informed they were playing against an AI. And developing such an AI is a nice IT project (at least, from my point of view).

But here's the thing. Just a few days after this AI was announced in February 2024, I saw two accounts playing boop appear, behaving very strangely. One of them flew somewhat under the radar, the other wanted to show off.

Before explaining what clues arouse my suspicions, let me provide a little context. My name is Florian Richoux, I'm a French AI researcher working in Japan. I actually developed an AI myself to play boop, but its purpose was scientific: it aimed to propose an innovative combination of two AI methods, not to create an agent that mastered the game perfectly. I talk about this project on this forum here: https://boardgamearena.com/forum/viewtopic.php?t=33790.

When this new account playing boop and crushing everyone appeared, I wasn't the only one suspicious. Other strong players shared my doubts. Then, in June 2024, I received this email from the BGA team (I know them a little, French connection).

(Dark theme for the win)

For those who don't read French, it says:
Hello Florian,
Have you shared the source code of your work? I have a player who is suspected of using an AI to win all the time. I don't know if that's possible? Is it possible to detect the use of your AI? <link to player ID>
Thanks,
Aurélien

This is just to let you know that it's very real, that it's not just me who's delusional, and that it naturally worries BGA.

Let's finally get to the heart of the matter: how do you spot players using AI? Keep in mind that you never have absolute proof, only an accumulation of strange facts. The more such facts you observe, the less doubt there is. Let's list the main ones:
- The player is new and surprisingly skilled. This is, of course, the first thing that catches your eye, but it's far from sufficient: some players are naturally talented, while others may have extensive experience playing elsewhere before coming to BGA. This fact alone is not enough.
- Look at these players' first games. Of the four accounts I've identified over the past two years, three had absolutely terrible first games, playing absurd moves, as if their AI wasn't yet well-trained. They stopped for a few days, then came back beating experts, reaching a level impossible for a human player starting from such a low skill level to achieve in such a short time.
- Even the best AI sometimes makes absurd moves. Regarding the account that worried the BGA team, this person had only lost twice, but always in the same way, in a rare situation that a strong to average player could easily understand. The AI ​​was clearly not well-trained for this situation and played a fatal move that a human player of that skill level wouldn't make, at least not twice.
- People using AI can sometimes play very fast at all times. Even a very strong player needs time to think about their moves during a high-level match.
- People who use AI to win do so because they crave recognition. They will tend to boast and show off. And sometimes, writing low-level or even erroneous strategic comments on this forum.
- Very often, they focus solely on the game they cheat on. They hardly ever play other games, and have even often opened a BGA account specifically to play that game with an AI.

Now, how should you react when you play against someone you suspect of using AI? I don't have any specific advice to give on this; everyone reacts as they see fit. But I can tell you how I react:
- I don't report them publicly, neither on the forum nor anywhere else. After all, strong suspicions aren't proof; I could always be wrong about the person. I do, however, talk to my boop-playing friends on BGA privately to get their opinion.
- I also don't report them to BGA. It's a small team (I don't know the exact number now, but recently there were only 10, half of whom were technical staff), they have other fish to fry, and it's a miracle that such a small team can maintain a platform with several million accounts.
- I don't play against the person. If I ever run into them in a tournament, so be it, I lose on time. I block them by giving them a thumbs down to minimize interaction. However, please don't abuse the thumbs down rule; don't be like those pathetic players who block strong players to avoid arena matches and climb the rankings by only playing easy games. This ridiculous attitude is everywhere, even among a so-called Master of this game, who clearly isn't skilled enough.
- I'm patient. Players using AI get bored once they've gotten what they're looking for, which is a bit of recognition and visibility by being ranked first for a while. After a while, they get fed up and stop playing. After all, it's not very fun to let an AI play for you...

[Ceaseless]

I really like this game and its community of players on BGA. I'm sad to see strong players stop playing, but this loss is compensated by the constant arrival of new, talented players.

Too dead for me these days. If real time play ever revived, then maybe I'd take a look again. Kind of wish I had a way to play against bots at least.

- The player is new and surprisingly skilled. This is, of course, the first thing that catches your eye, but it's far from sufficient: some players are naturally talented, while others may have extensive experience playing elsewhere before coming to BGA. It's necessary, but not enough.

Why is this necessary?

- I also don't report them to BGA. It's a small team (I don't know the exact number now, but recently there were only 10, half of whom were technical staff), they have other fish to fry, and it's a miracle that such a small team can maintain a platform with several million accounts.

I say go for it, especially if they're asking you anyways. There's always other fish to fry, but this fish can still make the list too.

However, please don't abuse the thumbs down rule; don't be like those pathetic players who block strong players to avoid arena matches and climb the rankings by only playing easy games. This ridiculous attitude is everywhere, even among a so-called Master of this game, who clearly isn't skilled enough.

I have a pretty good idea who one of them are. Constantly cancelling whenever they got paired against a passably rated player, was wondering why they'd go to Arena if they were just going to hide, but I guess if you block the users that would do it too. The block feature is largely... shady, frankly. At least as far as its gameplay lockdown goes. Guess there's another I didn't see enter Arena too... Wouldn't surprise me if there were others that have entered after my time.

[Kaepo]

- The player is new and surprisingly skilled. This is, of course, the first thing that catches your eye, but it's far from sufficient: some players are naturally talented, while others may have extensive experience playing elsewhere before coming to BGA. It's necessary, but not enough.

Why is this necessary?

Sorry, it was unclear, I edited my text. This phrasing comes from my scientific background: "a necessary but not sufficient fact."

[MarianNyah]

Having read this article, I've decided to come out clean: I am indeed using a software I developed to play my boop. games.

I acknowledge that I should've announced it on the forums and also at the game chat. That was what I did when I was testing my Checkers program at a certain website. However, inspired by the story of how DeepMind let their AlphaGo play games in secret against top Go players, I decided to try a different approach.

The AI, based on MCTS, is not neural network-based; it uses handcrafted weights and search techniques that took inspiration from Checkers, Go, and Gomoku. I also opted for a Thompson Sampling-based policy instead of the typical UCB one in an attempt to have the program quickly converge on the best move. There are other things happening under the hood which I will not discuss here. The result is as you've seen in our game (Kaepo vs MarianNyah game): the program is quick to capitalize on minor mistakes, plays all over the board, and obtains an almost uncontestable positional and material advantage.

Some might say it's just Ai AI. I, too, agree that some of the moves my program plays can be replicated by the AI in Ai Ai. Upon testing, my program can play against a 2-minute think time Ai Ai bot and win (it uses a think time of 3 minutes). Why the time difference? Ai Ai has a feature that displays how many positions it can process per second. The log said Ai Ai can simulate at least 55,000 games per second; my bot can only simulate at least 260 games per second. That's a 212x difference in speed!

The primary motivation for using my bot to play games is to see how good of a bot I can make. I saw the AlphaBoop post prior to making a BGA account and was intrigued: how well does this agent played? I was saddened to see that the AlphaBoop account does not regularly play games, so I changed the goal: figure out how well it plays against the top (human) players on the site.

The results came in quick. Human players appear to have a very "stiff" style when it comes to play. They focus on booping enemy pieces away, sometimes at the expense of their own development. The bot quickly won against those players. Some, however, had impressive playing styles. There were at least 3 times when human players forced my bot to make a blunder. One game in particular had me staring at the evaluation go down from 57% chance of winning to 40, 37, 35.

I knew I had to stop at some point. I recognize that what I'm doing is wrong and harmful to people who just wants a relaxing game. But there are people who, despite having "destroyed" by the AI, say "GG" at the end of the game. It might be just politeness, or just a reflex response from the losing side, but I couldn't imagine anyone hating the game they just witnessed say that phrase when they don't actually mean it. At least 3 players I have played with showed an improved playing strength after playing against my bot in Arena. One of them I feel like they're trying to emulate how my AI played. I see this phenomenon as people's willingness to learn how to play the game, try something new, and explore possibilities. And that some actually enjoyed the game.

One part of the post I feel like is an attack to me. I have to respond to that. I did post a strategy against your 1. b4 that tries to nullify the alleged advantage the first player gets after 1... c4 2. d3 by suggesting the 2... c4 response. Of course, your 2... d4 is also a valid response. Ai Ai and my AI both endorse this move. Both are AIs that the average human player cannot beat. I don't know how it could be an erroneous line, unless you can find a refutation. I've laid out the sequence as dictated by my program and the final position can be evaluated by anyone who has access to a board - basically everyone on the site that has an opponent matched. The position is a chaotic mess with Gray (first player) having to play a lot of edge moves just to secure a good position, something I know you will have a hard time playing given that your move statistics indicate that you almost always play at the center (or close to it) and barely play moves on the edge. This is also why I recommended it.

[MarianNyah]

I understand that you made this post with good intentions. However, certain words just don't sit right with me. So what if someone plays a little stronger than their rating? Maybe they have a prep line for a tournament or arena, or they're just creative, or they saw someone play the line and decided to copy it. I wish to protect the players that might get a lot of hate just because they play a little different. In fact, I want to encourage difference. By difference, I mean unique playstyles - using programs to win games is obviously wrong and I'm guilty of that. When someone challenges the status quo, these people give others the chance to re-evaluate their playstyle and learn which moves work and which doesn't. By your logic, a 38 rated player who plays a lot of odd moves are candidates for cheater behavior.

You say you love the game. I love it too, and I want to contribute something new by exploring new strategies (with my program). However, I feel like this post has a hidden agenda of gatekeeping "playing strong moves" to the top-ranked players.

As of writing, I have conceded all my tournament and arena games. I cannot seem to do the same to our game. The website tells me to finish it. If anyone wishes to play against my program, they may do so by messaging me and sending a friendly game invite. But I will not be participating in future tournament, arena, or any ranked game out of consideration for fairness. The SmolCatgirl account (which I manage to some extent) has also done the same.

I apologize to the boop. community for my actions. I should've done better.

p.s. the 1. b4 opening is not really an opening meant to dominate the center. it actually almost always leads to a messy game (if you let bots play it). it is also a very sharp opening along with a6, b6, c6, and all the other "strange" openings you can do as Gray (first player). if you want to dominate the center, just immediately play at the center (1. c4 or its other variants - d4, c3, d3).

[Kaepo]

Having read this article, I've decided to come out clean: I am indeed using a software I developed to play my boop. games.

I acknowledge that I should've announced it on the forums and also at the game chat. That was what I did when I was testing my Checkers program at a certain website. However, inspired by the story of how DeepMind let their AlphaGo play games in secret against top Go players, I decided to try a different approach.

It's good that you responded to this post; you made the right choice. It certainly wasn't easy and requires a certain amount of courage.

I understand why you did that. It was pretty clumsy of you (and that move was stupid of Deepmind), but since you shortened the experiment, there wasn't too much harm done.

The AI, based on MCTS, is not neural network-based; it uses handcrafted weights and search techniques that took inspiration from Checkers, Go, and Gomoku. I also opted for a Thompson Sampling-based policy instead of the typical UCB one in an attempt to have the program quickly converge on the best move. There are other things happening under the hood which I will not discuss here. The result is as you've seen in our game (Kaepo vs MarianNyah game): the program is quick to capitalize on minor mistakes, plays all over the board, and obtains an almost uncontestable positional and material advantage.

I would love to discuss this in more detail, in particular, to what extent Thomson sampling is more suitable than UCB here. Here is maybe not the right place to discuss that, though, but if you have more insights about that, I am curious to know more.

I knew I had to stop at some point. I recognize that what I'm doing is wrong and harmful to people who just wants a relaxing game. But there are people who, despite having "destroyed" by the AI, say "GG" at the end of the game. It might be just politeness, or just a reflex response from the losing side, but I couldn't imagine anyone hating the game they just witnessed say that phrase when they don't actually mean it. At least 3 players I have played with showed an improved playing strength after playing against my bot in Arena. One of them I feel like they're trying to emulate how my AI played. I see this phenomenon as people's willingness to learn how to play the game, try something new, and explore possibilities. And that some actually enjoyed the game.

Well, no, this isn't the right way to stimulate creativity around boop, I hope you realize that. But I completely agree with you that an AI can greatly help stimulate that creativity. I had the opportunity to speak with a professional Go player in Japan who told me he spent all his mornings studying AlphaGo games. A strong AI can certainly stimulate a community, but not by being hidden behind regular BGA accounts. Why not making an app/web site where anyone can play against your bot?

One part of the post I feel like is an attack to me. I have to respond to that. I did post a strategy against your 1. b4 that tries to nullify the alleged advantage the first player gets after 1... c4 2. d3 by suggesting the 2... c4 response. Of course, your 2... d4 is also a valid response. Ai Ai and my AI both endorse this move. Both are AIs that the average human player cannot beat. I don't know how it could be an erroneous line, unless you can find a refutation. I've laid out the sequence as dictated by my program and the final position can be evaluated by anyone who has access to a board - basically everyone on the site that has an opponent matched. The position is a chaotic mess with Gray (first player) having to play a lot of edge moves just to secure a good position, something I know you will have a hard time playing given that your move statistics indicate that you almost always play at the center (or close to it) and barely play moves on the edge. This is also why I recommended it.

There is a confusion here. When I wrote "writing low-level or even erroneous strategic comments on this forum", I wasn't thinking about you at all. I was thinking about that bragging 2024 account, who posted a couple of short-but-erroneous messages on this forum.

I read your comment on my opening post soon after you posted it, and at the moment I didn't fully understand it so I said to myself I will re-read it later when I have some peace and quiet. Then I forgot. ^^' But I am open to discuss it of course (otherwise, why would I take time to make a forum post about it?), as well as I am very open to criticisms on the b4-opening.

PS: I un-thumbs down on both of your accounts.

[GiannisIsTheBeast]

Are one of these AI players from Antartica, wears sunglasses and was an expert from the very start?

That one hasn’t played in a long time but I thought it was impressive they only lost twice and beat the current top ranked guy almost every time.

[Kaepo]

Are one of these AI players from Antartica, wears sunglasses and was an expert from the very start?

That one hasn’t played in a long time but I thought it was impressive they only lost twice and beat the current top ranked guy almost every time.

That was the braggy one, yes.

[Mononoke_Soushi]

TIL that even a silly little game about booping kittens has a bot problem.