LLM-Based Players

[Hervor]

That means it wasn't given any strategy advice but simply told to play the game a huge number of times and find out when it happens to win against itself, gaining an immense understanding of game positions and strategy but not necessarily of tactics or logic, right? So would an LLM just try to find a legal move by parsing the rulebook?

LLMs don't really have logic like that. If we take chess as an example, a chess bot (like the ones that have been around for ages) will calculate all possible future moves giving the current game setup and chose the one with the highest win probability.

A LLM will break down the current situation, assign it a token and check what other tokens are usually associated with this token and pick one of those.
Say white opens E4. An LLM will look at this situation and say "oh yes, this is 12345 and in all my data 12345 was followed by 23456 X% of the time so I will select 23456" and then move to C5, because the Sicilian Defense has been assigned the token 23456. But the LLM din't calculate whether this is actually a good move.

Could a LLM with a big enough data set be a good chess player? Sure, but a classic chess bot will do the job much more efficiently. LLMs are really good at certain tasks, but people want to apply them to situations where they don't make sense.

[stillframe]

If bot players are allowed, there should be certain rules, including at a bare minimum:

bots must always be clearly identified as such, including a standard bot avatar, a tag in their username, and description in profile
bots can only join a game by human invitation
bots are red-thumbed for all users by default (humans must manually green-thumb)
bots playing against each other is forbidden (increases server load)
bot games never rated
bots banned from arena

Even then I have concerns. I fear a slippery slope to bot infestation. Keep BGA first and foremost for people.

[lefalaf]

Plus there is the environmental cost. LLMs are wildly inefficient and resource heavy, and with little benefit, and poor accuracy. There are enough things harming the environment and driving up energy prices as-is. We do not need to accelerate that with more LLM demand.

(This is vastly different from purpose built non-LLM "AI" tools that help doctors find cancers, for instance, and are more efficient, benefit society, have proven results, etc.)

[Andrei3009]

I agree with the limitations presented by Stillframe. Ai will be good as a coach with whom you can play at any time and learn new things. The existence of such programs can raise the threshold of entry into the game, but it raises skill level of the game and potentially increases the replayability by opening up new ways for people to play.

However, such bots require a significant investment of time and effort (and money), and it is unclear who will do this.

[cigma]

I've gotten good at building with AI with the help of Claude Opus, and there are plenty of free models.

What's more, I believe that such a project is a responsability that needs more than only one developer to support it. Who will do this, if you can't do it for whatever reason? For example, if there is a serious bug when you are on vacation? Even when you think you have finished for a game, over time you may have to adapt to changes in the game itself or in the technical environment. For the same reason I think it's important to comply with standards and have a good documentation.

[Pocc]

I've gotten good at building with AI with the help of Claude Opus, and there are plenty of free models.

What's more, I believe that such a project is a responsability that needs more than only one developer to support it.

Why? There are plenty of open source projects that have a single developer. SQLite has a single developer, and it powers the databases apps use on your phone.

Who will do this, if you can't do it for whatever reason? For example, if there is a serious bug when you are on vacation? Even when you think you have finished for a game, over time you may have to adapt to changes in the game itself or in the technical environment.

Windows XP, for example, is no longer maintained, but that does not mean that hundreds of millions of people did not get value out of it. So I would push back on the idea that because something cannot last forever, that building it does not have value for the people who use it.

Taking a step back, the purpose of BGA is to provide a gaming platform for human players, and whether or not LLMs degrade won't affect that core purpose either way. LLMs can improve the experience by tutoring players to become better or providing a 3rd player for a 3 player game.

For the same reason I think it's important to comply with standards and have a good documentation.

Documentation is trivial. The question I have here is approval. I haven't heard back from the BGA team, but I'm sure the discussion here will inform their opinions.

[lefalaf]

Even then I have concerns. I fear a slippery slope to bot infestation. Keep BGA first and foremost for people.

Agreed. I hope the BGA admins hold the line on keeping LLM's out. Too many things are infested with it now.

I can't wait for this LLM/"AI" bubble to finally burst.

[Ingoschi]

Are there any examples yet where a LLM-based bot plays a boardgame better than e.g. a rule-based?

I found an interesting paper about using multi-agent LLMs to play Catan:
Agents of Change: Self-Evolving LLM Agents for Strategic Planning
https://arxiv.org/abs/2506.04651

[lefalaf]

From the conclusion:

Despite strong results, several limitations hindered performance from improving further. First, we evaluated players solely with win rate and final victory points, coarse metrics that sometimes mask subtler strengths and weaknesses. Second, the LLM occasionally hallucinated code or heuristics, requiring additional filtering. Third, the system was expensive to run due to inference costs, restricting the number of trials. Finally, performance remained closely tied to the quality of the underlying model, with more capable backends producing stronger players. Even with these constraints, HexMachina was able to autonomously induce an API, evolve a robust player, and achieve parity with AlphaBeta, the strongest human-crafted bot.

[lefalaf]

My read/skim was that this was primarily LLMs creating a player, not LLMs actually playing the game.

Although they did try out an LLM player and found that to fail. In the chart which follows, the win rate for that was 16.4%.

LLM Player.
We also evaluate a Reflexion-style agent (Shinn et al. (2023)) that reformats the game state into text and queries Claude 3.7 once per turn with a high-level goal. This baseline reflects the "prompt-centric" paradigm: the LLM directly drives play without any compiled memory or artifact reuse. Due to inference cost (approx. 70 queries per game), we limited this evaluation to 20 games with a model we had free access too, but it provides a critical comparison to show how quickly context saturation and lack of persistence hinder long-horizon play.

From the conclusion, it sounds like the LLMs needed human input and handholding along the way to correct for its lack of intelligence when creating the players. Did they bias the results with this additional human input? I have no idea, but presumably.

The players created by the 3 models (with human help) got to win rates ranging from 38.4% to 54.1% in 2P games it appears.