Board Game Arena, Catan, virtual dice and randomness - a short trial

[muntzer]

The idea of looking at the BGA implementation came from this thread:

"After 20 times playing, I am done with the BGA version [of Catan]"
viewtopic.php?t=26265

I played it 20 times, and I can safely say it's broken beyond being able to enjoy the game.
The dice generator is totally whacked out. 40-50 minute games, 60-70 rolls per game, and only five or six 7s rolled? In what universe does that occur?
Until it gets fixed, I'm done with Catan here. I'll stick to the analog version and deal with rolls that reflect the reality of statistics and probability.

This prompted numerous replies, many in a pretty unfriendly and condescending tone. Some relevant points were made, in general terms, about the common misunderstandings people have about the nature of random events.

Some common themes emerged, to which I've added a few observations of my own:

Randomness is "lumpy" - you shouldn't expect an even distribution of numbers in a small sample size.
For instance, it's quite likely that you'll see a run of 4 heads or 4 tails when you toss an unbiased coin ten times.
(When I wrote this, my first four coin tosses all came up heads, go figure.)

Independence of random events - no matter how often red has come up on a roulette wheel, the chance of red occurring on the next spin is exactly the same. It still feels weird, though, when we witness a long run of reds. In the same vein, how many 7s in a row might we expect to see, at least now and then?

We have monkey brains - we aren't very good at estimating probability. That is, the probability of random but structured outcomes is often counterintuitive. Take, for instance, the so-called 'birthday paradox'. If you take a room with 25 people who have no connection to each other, there's a better than 50-50 chance that at least two people in that room will share a birthday. This fact seems implausible, given 25 is such a small fraction out of the 366 possible birth dates that each person could have.

Confirmation bias - if we believe something to be true, we'll be more likely to notice and remember events that confirm that belief
and much less likely to notice the (possibly greater number of) events that contradict that belief. In a board game, when we are hanging on every roll, this confirmation is heavily amplified - we feel rising frustration when rolls that benefit our opponents outnumber those that benefit us.
We feel much less emotion when the converse occurs, so we are much less likely to remember it. "This @#$# game always cheats me!"

Risk aversion - We are evolved as a species to be more sensitive to negative events than positive ones, i.e. losing a $20 bet has a greater emotional weight than winning a $20 bet. This is why gamblers often feel so irrationally driven to recoup their losses.

The Dunning Kruger effect - people who don't understand probability are more likely to believe that (i) it's actually simple and (ii) they have a good understanding of it. In general, the less we know, the less aware we are of how much we don't know. Ironically, this effect is just as pronounced in people who are of above-average intelligence but haven't studied a discipline in detail. Hello to all those 'armchair experts' out there.

[muntzer]

Having a background in Pure Mathematics (which included probability and combinatorics, i.e. counting all possible arrangements of things), I was going to weigh in but stopped to reflect - what if the BGA dice really ARE flawed and not genuinely random? What amount of 'lumpiness' in dice rolls should we expect and accept? What sort of outcomes should we view with suspicion, i.e. things that shouldn't happen in 99.9% of games of Catan?

Luckily, the range of 'expected outcomes' can be calculated exactly using the binomial distribution. These calculations (close to the 'normal distribution' for large sample sizes) are relied upon in countless fields, from medical research to opinion polls, to casino managers and bookmakers to professional backgammon players.

I don't need to explain the mechanics of Catan here - the rules are simple and easily understood. Players will favour placing a settlement next to hexes with 5, 6 and 10 compared to a position next to hexes with 2, 4 and 11 (all else being equal). Yes, having the option of monopolising a resource, having access to a strategic trading port, or simply blocking an opponent's path can affect these choices. But the first player to place a settlement will always look for hexes with 5s, 6s, 8s and 9s vs 3s, 4s, 10s and 11s.

Catan is an influential but flawed board game. Its fundamental weakness is the unavoidable "lumpiness" of dice rolls. Your strategy might be brilliantly designed, but you may still be forced to look on helplessly as your hexes fail to produce anything for dozens of rolls while your opponents are busy constructing networks of roads, upgrading their settlements and buying development cards in bulk. There is literally nothing you can do. You can't trade your way out of the problem. Even the best Catan player in the world will be beaten by a novice if the novice lucks out with the right dice rolls in the early stages of the game. You can effectively corral a player into a small section of the island and maintain your resource advantage for the rest of the game.

It's for this reason that my board games group, after many enjoyable nights playing Catan, have effectively abandoned it. There are too many resource management / trading / development / engine building games that have an element of randomness but will never leave a player twiddling their thumbs for several (or many) turns in a row, as an inherent consequence of the game mechanics.

Yes, I know - you should avoid concentrating your settlements on a limited range of numbers, but then, we all know the pain and 'unfairness' of having the robber on your 6 or 8 - or whatever - when that roll comes up several times just when you really, really, really need that resource to advance your strategy. You sit there steaming when someone blocks your road or takes the spot you're one resource away from settling. And you may never roll a 7 to return the favour, or only when it's way too late. You get behind and stay behind. Game over. On the flip side, I've won three-handed games of Catan 10 to 4 to 4 with competent strategy and great rolls.

The inherent "lumpiness" problem in Catan will be, of course, even more problematic IF the BGA dice are prone to producing skewed distributions of rolls. Lopsided sets of dice rolls make a flawed game even worse.

To illustrate, let's imagine one possible scenario. What would happen if the BGA dice always favoured two of the possible rolls? (Out of the result space of 2-12.) For instance, in your next game of Catan, 5s and 11s occur in half of all rolls (instead of the expected frequency of 4/36 + 2/36 = 6/36 = 16.7%). Clearly, anyone lucky enough to build next to the 5 and 11 hexes would receive a huge advantage - the others would be picking up the scraps.

Of course, if 5s and 11s always came up too often, the flaw would be obvious. If, on the other hand, the two favoured numbers were different from game to game, this would be harder to spot. It might be 3s and 9s in one game. It might be 4s and 10s in the next game. Across all games of BGA Catan, we might not see any problems - 5s show up in 11% of rolls, 11s show up in 5.6% of rolls, and so on. But BGA Catan might still be prone to highly lopsided roll counts within individual games. It might be a broken implementation.

If that were indeed the case, strategy would become overwhelmed by sheer luck. If 5s and 9s came up half of the time in a given game, players who built next to 4s, 6s, 8s and 10s would essentially become spectators, despite there being absolutely nothing wrong with their gameplay. Even smaller skews in dice rolls could still result in frustratingly uneven games.

[muntzer]

So, that's the question at hand. Within a given game of BGA Catan, do highly uneven sets of dice rolls occur more often than we should expect? (When compared to mathematical models of randomness.) What are good criteria for deciding when a given set of rolls goes outside the 'normal' range of unevenness? We need an alternative to our in-built sense of fairness - as already observed, our intuition can let us down. Perhaps our concerns are actually more due to sour grapes / confirmation bias / faulty expectations?

My trial was simple - to look at my next 20 games of BGA Catan. One of the hallmarks of good research is to set start and end points for your trials before doing anything. Otherwise, you might be tempted to stop your trials when the data suits you. Or, even worse, you can look backwards to cherry-pick patterns in the data after the fact. If you scrutinise any random data set, you'll be able to find *something*. Random doesn't mean smooth. You might determine that more car accidents happen on calendar dates that are divisible by 5, but this finding won't stand up when someone analyses a new data set. Real-life research has been found to lean on these dishonest practices - so-called 'p-hacking' - drawing a hypothesis from selective data interpretation rather than starting with a clear hypothesis and testing it rigorously.

The maths of independent events is not hard. A finite number of trials, each having the same probability, can be modelled precisely with the
well-known binomial distribution. And you can do all the maths in Excel with the BINOM.DIST function.

Let's say your trial counts the number of sevens that come up when you roll two dice ten times. You want to know how likely it is that you will end up with three or fewer 7s in a given trial. In Excel, you can calculate this with the formula BINOM.DIST(3, 10, 1/6, TRUE). In this example, 1/6 is the probability of the individual event - rolling a 7 with two dice. TRUE means we are interested in the cumulative distribution - up to 3 sevens being rolled (i.e. 0, 1, 2, or 3 sevens). The Excel formula returns 0.9303 or 93.03%. In other words, if we ran this ten-roll trial 100 times, we'd expect to see 3 or fewer sevens most of the time - that is, in about 93 of those trials, and 4 or more sevens in only 7 of those trials, give or take a few. Clearly, both outcomes (3 or fewer sevens, 4 or more sevens) are plausible - one outcome is more likely than the other, but with a single trial, you shouldn't be shocked to see four or more sevens rolled.

In my actual trials, the actual number of dice rolls per game was almost exactly 100 (1,999 rolls in 20 games = 99.95 per game). The number of rolls per game was clustered very tightly between 97 and 102 rolls: 98, 101, 101, 102, 101, 101, 98, 99, 99, 101, 97, 100, 101, 99, 100, 99, 100, 101, 100 and 101.

Given this, we can start calculating expectations for how often each number (2,3,4..., 12) should come up per game.

For instance, we know that P(rolling a seven) is 6 in 36 = 1 in 6 = 16.7%. So, we should observe an average of about 17 sevens rolled per game of Catan. Sometimes it will be 15 sevens, sometimes 19 sevens, but over a long enough set of games, we should see the number of sevens average out between 16 and 17 per game. In my 20 games, sevens came up 16.46% of the time.

That's a pretty basic result - not very interesting. What's more useful is to develop a 'confidence interval' for how many 7s we see in a given game of Catan. While 16 or 17 is highly expected, it would be genuinely bizarre if 0 sevens were rolled or 50 sevens were rolled. Both outcomes are logically possible but still way outside reasonable expectations. What sort of range is reasonable to observe, then?

[muntzer]

To find this 'reasonable' range, we set a somewhat arbitrary figure that is less than but fairly close to 100%. This might be something like 95% or 99% or 99.5%, depending on how confident we want to be about the expected outcomes. A 95% confidence interval tells us what to expect in 19 out of 20 games, 99% for 99 out of 100 games, 99.5% for 199 in 200 games, and so on. The closer we get to 100%, the wider the range of expected outcomes will be.

So, let's start with a 95% confidence interval. This set of outcomes should apply to almost all games - 19 out of 20. It's a pretty decent 'rule of thumb' - if we spend a few hours playing BGA Catan, we shouldn't expect outcomes outside the interval. Not impossible, but unlikely.

To find this confidence interval, we use the Excel BINOM.DIST function to find two values:

For what value of x1 does BINOM.DIST(x1, 100, 1/6, TRUE) = 2.5% (x1 will be less than 16.7, i.e. lower than the avg.)
And for what value of x2 does BINOM.DIST(x2, 100, 1/6, TRUE) = 97.5% (x2 will be more than 16.7, i.e. higher than the avg.)

The interval we find will therefore have 2.5% of outcomes from x1 down and 2.5% from x2 up. The 95% range lies between x1 and x2.

When you plug this formula into Excel, the values that get us closest to 2.5% and 97.5% are x1 = 9 and x2 = 24. That means our 95% confidence interval contains the values above x1 and below x2 - i.e. the range 10 to 23. More precisely, Pr(10 ≤ x ≤ 23) equals 94.1% - so, slightly less confidence than our target level. A bit of jiggling with the boundaries gives us a range that is closer to 95%: Pr(11 ≤ x ≤ 26) = 95.1%

Either way, those are pretty wide confidence intervals. Basically, if you see somewhere between 10 and 24 sevens rolled in a game of BGA Catan, you should shrug and say, "yeah, that's cool". You'll definitely notice both ends of that interval, though. Only 10 sevens out of 100 will 'feel' like the robber has gone on holiday. 24 sevens will 'feel' like the robber is lying in wait around every corner; you'd see several sevens rolled in a row at some point. AAAARGHH!

But, yeah - both sorts of games can be expected every now and then. We should expect to see 10 (or fewer) sevens rolled in 4.3% of games and 24 (or more) sevens rolled in 3.8% of games.

It's when we go outside these ranges that things start heading into 'Twilight Zone' territory. Seeing no sevens rolled in a game of Catan is a one-in-a-million scenario - actually, one in 82,817,975. That's about once in 7,000 years of continuous play. As we head back towards our confidence interval, the odds increase.

3 or fewer sevens out of 100: once in 54,752 games (4.7 years of play)
6 or fewer sevens: once in 766 games (24 days of play)
9 or fewer sevens: once in 47 games (35 hours of play)

Going the other way:

25 or more sevens in 100: once in 46 games (34 hours of play)
28 or more sevens: once in 322 games (10 days of play)
31 or more sevens: once in 3,381 games (3½ months of play)
34 or more sevens: once in 52,434 games (4½ years of play)

Where you set the upper limit of expected sevens in a game of Catan is up to you, but seeing outcomes that should happen only once in months of continuous play is certainly pushing the friendship.

Here are the 95% confidence intervals for all the rolls (one game = 100 rolls):

2s: between 0 and 5 rolls per game (93.9% confidence)
3s: between 2 and 10 rolls per game (95.4% confidence)
4s: between 4 and 14 rolls per game (95.3% confidence)
5s: between 6 and 17 rolls per game (94.5% confidence)
6s: between 8 and 21 rolls per game (95.7% confidence)
7s: between 10 and 24 rolls per game (95.7% confidence)

(Due to symmetry, the confidence intervals for 12s are the same as for 2s, and so on.)

Note: checking these sorts of confidence interval calculations is as far as I'm going. You can also analyse patterns of rolls within a given game, i.e. how often does the same number come up three times in four rolls? That sort of analysis requires compiling the history of dice rolls in exact order - well beyond my stamina or care factor. Be my guest. But - as a simple rule of thumb - we shouldn't expect to see the same number come up too many times in a row. The probability that you'll roll four 12s in a row during a game is about 1 in 17,300 (once in 18 months of continuous gameplay). If you see it happen, you are well within your rights to suspect something is busted.

(Note: all 12s in a given set of four rolls = 1 in 1,679,616, but there are 97 sets of four consecutive rolls in a game of 100 rolls, so the probability goes up by that factor.) Here's a useful set to work from:

Three 2s in a row: 1 in 476 games (15 days)
Three 3s in a row: 1 in 60 games (45 hours)
Three 4s in a row: 1 in 18 games (13 hours)

Three 5s in a row: 13% of games
Four 5s in a row: 1 in 67 games (51 hours)

Three 6s in a row: 26% of games
Four 6s in a row: 1 in 28 games (21 hours)

Three 7s in a row: 45% of games
Four 7s in a row: 1 in 13 games (10 hours)

(Again, by symmetry, these odds are identical for 12s, 11s, etc.)

[muntzer]

Trial results

All 20 games of BGA Catan were played in Arena mode with three players per game. Given I was ranked in the elite bracket (my tournament ranking was mostly between 400 and 600), almost all of my games were against other 'elite' players. I didn't keep track of how many I won, but my ranking did trend upward towards the end. I don't recall encountering any players in the 'super elite' category of 1700+ ELO. Scary people.

Game length

As already observed, there were almost exactly 100 rolls per game of Catan - all of them between 97 and 102 rolls.

Why so consistent? I'm not really sure, but perhaps the evenness of competition was part of the story. The final points totals weren't always close, but that doesn't mean one player was clearly playing better than the others. Points can swing very quickly as the longest road card swaps hands. Victory point cards can end up with one player by chance. Resource swings get bigger as more settlements and cities are built.

There may be simple maths involved - if resources gained per turn increase steadily over the course of a game, they eventually reach a point where one or more players can reach ten points without too much luck going their way. Add up all the resources, and you inevitably reach a certain number of settlements, cities and VP cards.

Between evenly matched players, games typically don't end quickly because Catan always turns into an arm wrestle. Good players anticipate and try to head off their opponents' strategies before they bear fruit. Anyone who seems to be in the best position (not necessarily the lead) will be targeted by the other players (unless they chicken out and start competing for 2nd place). In one game, I settled a sheep trading port and had settlements or cities next to four sheep hexes numbered 5, 8, 9 & 10. I was beaten miserably. Both opponents stole from me relentlessly and blocked me in every direction I tried to move. Precisely because I was the clear favourite to win.

Another interesting result: I would have expected that higher counts of 7s would produce longer games, but that simply wasn't the case, at least in my limited trials. The extremes of 7s rolled were 11 and 26 - both games were completed in exactly the same number of turns - 101. Was the game with 26 sevens tedious? Yes, it was. But no longer in terms of actual turns.

[muntzer]

The roll results across all 20 games:

The cumulative results from 1,999 separate rolls weren't surprising. They were very close to the progressive step-ups that are expected in the binomial distribution. (The numbers in brackets indicate how many they were above or below expected counts.)

2s: 62 (+6)
3s: 100 (-11)
4s: 161 (-6)
5s: 207 (-15)
6s: 291 (+13)
7s: 329 (-4)
8s: 270 (-8)
9s: 240 (+18)
10s: 185 (+18)
11s: 95 (-16)
12s: 59 (+3)

None of these variations lies outside the 95% confidence interval for 1,999 dice rolls. The closest to the edge of the confidence intervals were the 11s (four rolls above the lower limit) and the 9s (ten rolls below the upper limit). Of course, these are only totals across all 20 games. Things got much more interesting when I started looking at the dice rolls for individual games.

2s: Expected between 0 and 5 rolls per game, avg. 2.7
Actual: between 0 and 9 rolls, avg. 3.1 (+12%)
Number of results outside 95% confidence interval: four (6, 6, 7, 9)

3s: Expected between 2 and 10 rolls per game, avg. 5.6
Actual: between 0 and 10 rolls, avg. 5.0 (-10%)
Number of results outside 95% confidence interval: two (0, 1)

4s: Expected between 4 and 14 rolls per game, avg. 8.3
Actual: between 3 and 14 rolls, avg. 8.1 (-3%)
Number of results outside 95% confidence interval: two (3, 3)

5s: Expected between 6 and 17 rolls per game, avg. 11.1
Actual: between 6 and 16 rolls, avg. 10.4 (-7%)
Number of results outside 95% confidence interval: zero

6s: Expected between 8 and 21 rolls per game, avg. 13.9
Actual: between 8 and 23 rolls, avg. 14.6 (+5%)
Number of results outside 95% confidence interval: one (23)

7s: Expected between 10 and 24 rolls per game, avg. 16.7
Actual: between 11 and 26 rolls, avg. 16.5 (-1%)
Number of results outside 95% confidence interval: one (26)

8s: Expected between 8 and 21 rolls per game, avg. 13.9
Actual: between 9 and 20 rolls, avg. 13.5 (-3%)
Number of results outside 95% confidence interval: zero

9s: Expected between 6 and 17 rolls per game, avg. 11.1
Actual: between 4 and 28 rolls, avg. 12.0 (+8%)
Number of results outside 95% confidence interval: four (4, 5, 21, 28)

10s: Expected between 4 and 14 rolls per game, avg. 8.3
Actual: between 1 and 16 rolls, avg. 9.3 (+11%)
Number of results outside 95% confidence interval: two (1, 16)

11s: Expected between 2 and 10 rolls per game, avg. 5.6
Actual: between 0 and 10 rolls, avg. 4.8 (-14%)
Number of results outside 95% confidence interval: two (0, 1)

12s: Expected between 0 and 5 rolls per game, avg. 2.7
Actual: between 0 and 6 rolls, avg. 3.0 (+6%)
Number of results outside 95% confidence interval: two (6, 6)

I was surprised by these results - there are lots more outcomes outside our confidence intervals than I would have expected in this case. From 20 trials, I would have expected to see values outside those confidence intervals about once per number rolled; in other words, about 11 in total. What I found instead was 20 of those outliers.

[muntzer]

What is even more surprising is how far some of those results are outside the confidence interval.

An exemplar of this is the roll counts for 9s. In 4 out of 20 games, the number of 9s rolled was outside the confidence interval. But one of those games was staggeringly skewed - 28 nines turned up in 102 rolls. That's 11 more nines than the upper limit of our expectations. From our binomial calculations, we'd expect to see more than 28 or more nines (out of 102 rolls) only once in 230,177 games. In other words, if you played BGA Catan continuously for 19 years and 8 months, you'd only expect to see that many 9s once. But instead, it happened in the 4th game of my trial. And yes, I do remember that game very well. I had no settlements next to 9s; my opponents both did. It was a bizarre experience watching them both collect shedloads of resources while I got a few scraps here and there.

Perhaps you're thinking: "Well, odd things do happen. People win the lottery every week even though the odds are minuscule." Fair point. While the 28 nines certainly stand out as a bizarre result, several other results were also well beyond expectations.

Rolling 2s: Count of 9 from 102 rolls
Odds of event: 1 in 452 games

Rolling 3s: Count of 0 from 102 rolls
Odds of event: 1 in 340 games

Rolling 6s: Count of 23 from 100 rolls
Odds of event: 1 in 107 games

Rolling 7s: Count of 26 from 101 rolls
Odds of event: 1 in 74 games

Rolling 9s: Counts of 4 from 97 rolls, 21 from 99 rolls, 28 from 102 rolls
Odds of events: 1 in 75 games, 1 in 380 games, 1 in 230,177 games

Rolling 10s: Counts of 1 from 97 rolls, 16 from 101 rolls
Odds of events: 1 in 471 games, 1 in 107 games

Rolling 11s: Count of 0 from 98 rolls
Odds of event: 1 in 271 games

(Note: all of the odds above are calculated on a cumulative basis, not just that specific outcome, i.e. 28 or more nines from 102 rolls. In other words, I'm doing the maths correctly.)

With a confidence interval of 95%, we should see some outcomes beyond the expected range - but not this many, and not so many so far outside that range. Several observations should only be seen once every 300, 400, or 500 games, rather than several times within a sample space of 20 games.

[muntzer]

To make things worse, the combined counts for different sets of hexes were often ridiculously skewed, i.e.

Game #3: 3s came up once; 2s and 10s = 22 rolls
Game #4: 3s + 4s + 8s = 15 rolls; 2s + 9s + 10s = 46 rolls
Game #5: 5s + 8s = 17 rolls; 6s + 10s = 33 rolls
Game #8: 4s + 5s + 10s = 17 rolls; 6s + 8s + 9s = 52 rolls
Game #10: 10s + 11s = 2 rolls; 2s + 12s = 12 rolls
Game #11: 4s + 9s = 7 rolls; 3s + 11s = 18 rolls
Game #12: 5s + 10s = 10 rolls; 6s + 11s = 33 rolls
Game #15: 4s + 9s = 8 rolls; 2s + 3s + 5s = 32 rolls

That's 8 games out of 20 where the production from comparable hexes was out by factors of 2x or 3x, or even more. These results suggest that the random dice implementation in BGA Catan is, as the original poster suggested, "totally whacked out". Being on the wrong end of those production mismatches would be highly frustrating. Being on the right side, somewhat embarrassing.

But hey - what would I know?

As I said in the introduction, I didn't cherry-pick the data. I took 20 consecutive games of BGA Catan as they came, with no dodging or editing. I'm more than happy for anyone to look at my raw data and show me where I've gone wrong.

[Romain672]

Just to see how likely/unlikely all this is, I just did a document to generate 20*100 rolls.
It's present here:
https://docs.google.com/spreadsheets/d/ ... edit#gid=0


So you have seen 10 games which got some numbers of rolls outside the interval.
I will check that specific point.
Here is the number of rolls outside the interval you defined by using my doc 10 differents times:
11, 12, 13, 5, 10, 12, 19, 19, 16, 10.

And so if I did things correctly, your claim

With a confidence interval of 95%, we should see some outcomes beyond the expected range - but not this many, and not so many so far outside that range. Several observations should only be seen once every 300, 400, or 500 games, rather than several times within a sample space of 20 games.

look false.
I didn't checked any other point since it would require more work.

But from my understanding of your work, there was 11 differents weird number of rolls for a specific number on each game. There was 20 games. So you are looking for 220 potential weird outcome.
Then you pick a 95% confiance interval. So we expect around 1/20 of those to be outside of it.
Which leave about 11, which is totally what I and you get.

[muntzer]

But from my understanding of your work, there was 11 differents weird number of rolls for a specific number on each game. There was 20 games. So you are looking for 220 potential weird outcome.
Then you pick a 95% confiance interval. So we expect around 1/20 of those to be outside of it.
Which leave about 11, which is totally what I and you get.

No, you haven't read me correctly.

I was surprised by these results - there are lots more outcomes outside our confidence intervals than I would have expected in this case. From 20 trials, I would have expected to see values outside those confidence intervals about once per number rolled; in other words, about 11 in total.

What I found instead was 20 of those outliers.

You've also completely ignored how far some of those outliers are - 28 nines out of 102 rolls should only happen once in 230,177 games of Catan.
Please read my all of the posts above before telling me I'm mistaken.