Please update ELO tables for Hanabi

[HanabiGrandMaster]

Intro
Which variant is more difficult? Avalanche of colours? Tricky? Or basic five colour variant with blackpowder? All these options have 6 colours, but play differently? And should their difference in difficulty be reflected in ELO?

Currently ELO is a function of variant and number of players. It has been this way since 2014, and the tables haven't been updated for black powder. You can find the tables here:viewtopic.php?f=3&t=4814

I think it's time for an update of these tables, since they are not based on reality. I have a couple of proposals, that I hope that we can discuss.

Goals
I would like ELO to reflect skill.
I believe skill can be measured by looking at the ratio of perfect games vs non-perfect games.
I also believe that different variants have different difficulty levels.

Default skill levels in BGA
BGA has filters for certain skill levels based on ELO. They are:

• Beginner: 0 ELO points
• Apprentice: 1-99 ELO points
• Average: 100 - 199 ELO points
• Good: 200 - 299 ELO points
• Strong: 300 - 499 ELO points
• Expert: 500 - 699 ELO points
• Master: 700+ ELO points

My opinion
Based on experience, some simplified calculations, and simulations (source: https://github.com/Quuxplusone/Hanabi), I want to assume that it is possible to get perfect score in >90% of the decks, no matter what variant you play.

I think that in general, the difficulty varies as follows:

• Basic 5 colour variant: 100%
• Tricky, Avalanche, and 5 colour + blackpowder: 200%
• Difficult (5 multi's): 250%
• Seven colours: 300%

And I think playing without flamboyants doubles your difficulty (at least twice as likely to not-get-perfect).

I think 2-player-official-variant games are extra hard, and 5-player-official-variant games are also a bit more difficult than normal. Perhaps they should be ranked differently, but I would suggest to do so in a second iteration.

To match these difficulties with the BGA ELO-levels, I would suggest these tables:
5 colours

Code: Select all

score   ELO
0       0
1 - 24  100
25      330

6 colours

Code: Select all

score   ELO
0       0
1 - 29  100
30      550

6 colours - Difficult

Code: Select all

score   ELO
0       0
1 - 29  100
30      660

7 colours

Code: Select all

score   ELO
0       0
1 - 34  100
35      770

And everywhere where there is 100 ELO score, you can make it 200 ELO score without flamboyants.

This will result in
Beginner: Someone who has never played on BGA
Apprentice: Someone who doesn't yet gets perfect scores in basic variant
Average: Someone who gets perfect scores in <50% of the games in basic variant
Good: Someone who gets perfect scores in >50% of the games in basic variant, or someone who gets <90% perfect score with 6-colour-variants
Strong: Someone who gets perfect score in >90% of the games in 6-colour-variants
Expert: Someone who gets perfect score in >90% of the games in difficult-variant, or >70% perfect score with 7-colour-variant.
Master: Someone who gets perfect score in >90% of the 7-colour-variant games.

Additional thoughts
I've also noticed that multiple players are playing "difficult" and "unranked" only games since the ELO-apocalypse in order to preserve their ELO. When the ELO tables are updated, I hope the powers that be will also reset everyone's ELO.
This is the list of players I encountered that haven't played a ranked game in months:
Vikingattien (https://boardgamearena.com/player?id=63252593)
Mi-K-L (https://boardgamearena.com/player?id=89584016)
fuerdrake (https://boardgamearena.com/player?id=86979007)
Ermand (https://boardgamearena.com/player?id=84636151)
tomvetr (https://boardgamearena.com/player?id=84484071)
teobellu (https://boardgamearena.com/player?id=93910173)
ondiandreas (https://boardgamearena.com/player?id=33452221)
Rase-moquette (https://boardgamearena.com/player?id=90697364)

[Blacktango]

Hi, thanks for bringing this up!
I find it useful.

Regarding your suggestion, I think you meant 1-29 and 30 (instead of 1-24 and 25) for more than 5 colors.

Here are some thoughts/questions about your suggestion:

Difficult mode (5 multi ELO was off because it was considered too random). Maybe 250% is ok.

How do you combine Bk powder with Avalanche or with Tricky mode? Is it all 200%?

I guess you're just focusing on the difficulty and not on the complexity because you don't make the difference between Avalanche and Tricky? Since a more complex mode increases the number of bombs, we could say that Avalanche mode is harder to win than Tricky mode.

You base all your calculations on "perfect or not perfect". I think this is relevant for well-experienced players, but does it help to split players below expert level?
To get the same ELO by winning with 10 points and 24 points seems a bit extreme to me.

[HanabiGrandMaster]

Mistakes corrected
Thank you for pointing out my copy-paste mistake (I copied because it was in "code", and forgot to adjust scores). I've edited the tables to reflect the correct numbers.

Difficult variant
The difficulty with only five multicolour cards, is the lack of clue-tokens, and the chances of getting bottom decked.
Without blackpowder, there is a 1/60 chance that m1 is the bottom card (1.7%), and depending on how many players you have, it's probably also impossible to get perfect score with m1 being the second from the bottom card. And having five less "copies", means you're likely to play the game with 4 or 5 less clue-tokens.

Difficult variant with blackpowder
Since most games I play end a game with 6 - 8 tokens remaining, I don't think the clue-tokens are much of an issue - if you are an experienced player. When you add blackpowder in the deck, you reduce the odds of getting bottom decked to 1/70 (1.4%), which is ~16% less. Since you boost ELO rewards with +25% for this variant, I think the -16% is enough to offset the boost to just settle on the added difficulty of blackpowder.

Perceived difficulty
Over the last 6 months I've been asking players (long time players, but ranked from strong to master) what they found more difficult. In general, the strong players find avalanche the most difficult. Because, in general, they find it harder to stick to "good touch", so they constantly unintentionally mark the multicolour. Master players, in general, find blackpowder more difficult, since save-clues might unintentionally mark a copy powder card. And those unintentionally marked powder cards are hard to correct. And master players find, in general, avalanche the easiest variant to play, because they can abuse the multicolour to mark more collateral cards "for free" to boost clue-efficiency. So, in general, I think every advantage in the eyes of some players is a disadvantage in the eyes of other players. So, in general, I don't think there is much difference in the difficulty between the 6-colours variants. (Except for the only-5-multi's, as mentioned in the previous paragraphs.)

Answering BlackTango's questions
To be honest, I just pulled the 2x more difficult out of thin air. But it makes it so much easier to distinguish between beginners (who like to stick to 5 colours) and players who are ready for a more challenging game. Assuming people start with 5 colours only, you can progress from apprentice to good (1 - 300 points). When you get 80% - 90% perfect games, you will reach 270 - 300 ELO points, making you ready for a 6th colour.

To stay at 300 ELO points, you need to get 55% perfect games. Then you can improve to 80% - 90% perfect games, which makes you reach 450 - 500 ELO points (strong).

With regards to non-perfect games, I don't think there is a difference between getting 20 points, 23 points, or 24 points. With flamboyants, it's possible to get perfect games in >99% of the decks. When I get 24, it's usually because of getting bottom decked, and anything lower is caused by inexperience. I am happy to put my 1% bottom-deck games in the same bucket as inexperienced players, since 1% doesn't affect ELO significantly. I don't think non-perfect games should be rewarded at all, but I also feel that finishing a game is better than bombing out. Hence my suggestion to makes bombs worth 0 ELO, and all non-perfect finished games 100 ELO.

Currently, there are a lot of players in the master level that mostly play 7 colour variant, but only get 50% perfect games. Let's use this as an example and say their other games are 35% 34 points; and 15% 33 points. In my suggestion, they would be ranked around 400 ELO: dead in the middle of "strong". In the current ELO tables, players still get ca. 85% of the ELO points for getting 34 points. If you change my suggestion tables to include this 85% reward, then the example player would be ranked around (0.5 x 770 + 0.35 x 0.85 x 770 + 0.15 x 100 =) 629. This would bump him up to expert. But I think this player would be much weaker than someone who gets 94% winrate with difficult mode.

I think in practice, experienced players will get perfect, "perfect -1", or perfect -2" scores, or bomb out. Only with inexperienced players will you see lower scores. So I can see your point of view, where you would like to distinguish those scores, since they are common. But I hope you can see that those non-perfect scores should be (relatively) rare (1% - 10% for masters, 10% - 30% for aspiring masters) that changing these numbers is fidgeting in the margin. It would also send a wrong signal that getting a non-perfect score is acceptable.

What player would you find more skilled: A player who would get 34 points in 100% of their games? Or a player who gets 35 points in 85% of their games?

[Jellby]

I don't think non-perfect games should be rewarded at all, but I also feel that finishing a game is better than bombing out. Hence my suggestion to makes bombs worth 0 ELO, and all non-perfect finished games 100 ELO.

By the time you reach 900-1000, 0 or 100 makes no difference, so that would lead to intentionally bombing out games that cannot be perfect (or that the bomber thinks so).

Of course, with those changes there would be no 900-1000...

[Blacktango]

I think in practice, experienced players will get perfect, "perfect -1", or perfect -2" scores, or bomb out.

I really agree with this.

About the same reward for 1-24, isn't that discouraging?
If you are a relative beginner, you might feel that you have close to 0% chance of getting the perfect score. If you also know that the reward for 15 and 24 is the same, why would you try so hard to get 25 (or even 24)?
Why not target 1 point?

Of course, with those changes there would be no 900-1000...

Since you can't see who is 700 and who is 1000 ELO, it doesn't seem to be a problem to me.

[Travis Hall]

I agree that it would be good to update the way ELO is calculated for Hanabi. At the least, the current method has been rendered obsolete by the addition of Black Powder and 5 Flamboyants options. However, I disagree with some of what has been suggested by at least one person.

The game presents a wide array of possible outcomes. Even if highly-skilled players will tend to cluster their games at certain outcomes (perfect, perfect-1, perfect-2, and 0, perhaps) that should be respected. It is more difficult to score 29 than 28, and more difficult to score 26 than 25. The table might be weighted such that the effect of those differences on ELO is small, but it should still exist (as long as it is practical to implement).

Nobody scores 90% perfect in 4-player or 5-player games, and I really doubt anybody does in 3-player. Not even those whose rating is well above 900 in the current system. Possibly there should be some sort of correction; I have long been of the opinion that new masters are really not very good, so recalibrating so that the low-end of master becomes expert and only the high end remain master (and as a result being able to filter participation in games better) may be a good thing. However, that is not necessarily so useful if it becomes too hard to get together enough players for masters games.

The reason nobody scores 90% perfect in multiplayer is simply that you need to gather together 4+ players who are all that highly skilled; one poor player has much more effect on the game than one excellent player. More players in the game means more chances that somebody is having an off day. If I played with four clones of myself, maybe I’d be scoring perfect 90% of the time, but I do not have four clones of myself. You might replicate that with a team who play together consistently, and only play together, but I don’t think that is something BGA wants to encourage.

In 2-player, maybe you’d get some players scoring perfect 90% of the time. That’s also a very different game from larger games, though. Ideally it would be split into a separate rating, but I think we all know that will not happen. But I would hate to recalibrate on that expectation of 90% perfect to find that only 2-player only players can reach master.

Black Powder makes the game harder. Flamboyants makes the game easier. Rainbow as a separate suit is harder than non-rainbow; Avalache of Colours is harder again. However, the difficulty increase is not constant, not linear, not the same across number of players, and not the same at all skill levels. (I do not believe that Avalanche makes the game easier, but it does provide more opportunity for application of skill, so it has more effect on low-skill players than high-skill players.)

The number of players really does have an effect on difficulty. We should not make the mistake of thinking that the difficulty change all comes from the number of visible cards, the number of cards that can be held in hands, or the number of clues that can be given over the course of the game. The complexity of tracking clues, and having the entire team communicate effectively, become greater factors as the team size increases. There should definitely be adjustment for the number of players, but doing that adjustment well may require some fairly extensive data analysis. Theoretically you could build a system that self-adjusts based on BGA’s record of games, but whether BGA’s systems could handle it is another matter.

I believe that both 3-player and 5-player are more difficult than 4-player, but for diffent reasons. (3-player provides less resources, so needs better resource management. 5-player provides greater communication challenges.) I think the current system is way too harsh on 3-player compared to other player counts.

[HanabiGrandMaster]

The game presents a wide array of possible outcomes. Even if highly-skilled players will tend to cluster their games at certain outcomes (perfect, perfect-1, perfect-2, and 0, perhaps) that should be respected. It is more difficult to score 29 than 28, and more difficult to score 26 than 25. The table might be weighted such that the effect of those differences on ELO is small, but it should still exist (as long as it is practical to implement).

Non-perfect scores are not worth ELO points
Let's entertain just three possible scores: Perfect (X), Perfect-1 (Y), and Perfect-2 (Z).
Player A, on average, scores: {55% X, 30% Y, and 15% Z}.
Player B, on average, scores: {1% X, 90% Y, and 9% Z}.
Player C, on average, scores: {75% X, 15% Y, and 10% Z}.

What does your gut feeling say about which player is best or worst?

To me, it seems that player B is the worst, since they probably couldn't score a perfect game if their life depended on it. Yet, rewarding Y seems to justify player B's playstyle.

Player C seems like a better player than A. Do you think C is in "the same league" as A? I think they are not in the same league, so there should be at least 200 points of ELO difference between them. That means: 0.2X - 0.15Y - 0.05Z > 200. If X ~ 600, then Y < 0 and Z < 0, which is impossible. This is the reason why I think non-perfect scores shouldn't be worth ELO points.

I welcome discussion about my assumptions:
* When is someone "the same league"?
* How do you value almost-perfect scores?

Nobody scores 90% perfect in 4-player or 5-player games, and I really doubt anybody does in 3-player. (...)

I disagree with that statement, since "nobody" and "everybody" are too extreme.

In addition: if it is rare to meet players that have such high win-rates, does it mean it's not possible, or that people aren't trying hard enough? I think it is possible, and the ELO tables should reflect that.

The reason nobody scores 90% perfect in multiplayer is simply that you need to gather together 4+ players who are all that highly skilled; one poor player has much more effect on the game than one excellent player. More players in the game means more chances that somebody is having an off day. If I played with four clones of myself, maybe I’d be scoring perfect 90% of the time, but I do not have four clones of myself. You might replicate that with a team who play together consistently, and only play together, but I don’t think that is something BGA wants to encourage.

In 2-player, maybe you’d get some players scoring perfect 90% of the time. That’s also a very different game from larger games, though. Ideally it would be split into a separate rating, but I think we all know that will not happen. But I would hate to recalibrate on that expectation of 90% perfect to find that only 2-player only players can reach master.

I assume that perfect games are possible in >90% of the games. Let's say to get a perfect score, there is only one good move per turn that a player can make. Depending on the variant, there are ~60 - 100 moves per game. In a 2-player game, with 100 moves, each player has 50 moves to make.
The chance that player A makes a good move, is called Ma
The chance that player B makes a good move, is called Mb
So Ma^50 * Mb^50 > 90% (In fact, it should be closer to >99%, because ca. 10% of the decks is unwinable in 2-player games).

I would agree that 2-player games are simpler, since the amount of moves a player can make is quite limited compared to for example a 4-player game. However, 4-player games are often shorter (only 80 moves or so). So you would get:
Ma^20 * Mb^20 * Mc^20 * Md^20 >90%

If you assume that every player is more or less equally skilled (Mx), then Mx^100 > 99%, and Mx^80 > 90%, which would make Mx ~ 0.999 for 4-player games, and 0.9999 for 2-player games.

So, similar to the point I was making before: non-perfect should not be rewarded. I think it's possible to play perfect, and mistakes should be punished harder. If your perception is that 90% is not possible, is because you see too many player who make mistakes too often, but in the current system are still considered "master" level.

Black Powder makes the game harder. Flamboyants makes the game easier. Rainbow as a separate suit is harder than non-rainbow; Avalache of Colours is harder again. However, the difficulty increase is not constant, not linear, not the same across number of players, and not the same at all skill levels. (I do not believe that Avalanche makes the game easier, but it does provide more opportunity for application of skill, so it has more effect on low-skill players than high-skill players.)

Do you have some numbers? How much easier? How much harder? Why?

The number of players really does have an effect on difficulty. We should not make the mistake of thinking that the difficulty change all comes from the number of visible cards, the number of cards that can be held in hands, or the number of clues that can be given over the course of the game. The complexity of tracking clues, and having the entire team communicate effectively, become greater factors as the team size increases. There should definitely be adjustment for the number of players, but doing that adjustment well may require some fairly extensive data analysis. Theoretically you could build a system that self-adjusts based on BGA’s record of games, but whether BGA’s systems could handle it is another matter.

I believe that both 3-player and 5-player are more difficult than 4-player, but for different reasons. (3-player provides less resources, so needs better resource management. 5-player provides greater communication challenges.) I think the current system is way too harsh on 3-player compared to other player counts.

I agree with the increased complexity with the increased number of players. However, I think it barely affects the outcome. The increased amount of visible information offsets makes it easier to give clues. But the types of clues you can give increases. I think those two factors balance out. I also think updating the ELO tables should be an iterative process. I propose that in the first iteration we keep it as simple as possible, and make the number-of-players not a factor.

[Jellby]

If any change is done, I think it should be done based on hard (statistical) evidence and maybe some decisions about the desired outcome. Not on gut feelings (or worse, perceived trends and patterns) and opinions about what is harder or better.

[Travis Hall]

What does your gut feeling say about which player is best or worst?

Nope, not getting into those sorts of judgement calls. ELO is not a tool to shape player behaviour towards a particular style of play. The rules of the game ask us to strive towards a higher score, and the ELO rating calculations should reflect that, and not your opinion concerning the desirability of perfect scores, or whatever my opinion, contrary or otherwise, might be.

If here’s some hard data concerning how difficult it is to achieve particular scores, that might be taken into account (since we have no choice but to weight the scores somehow) but it is not for us to decide that players should be encouraged to seek perfect scores above all else.

Nobody scores 90% perfect in 4-player or 5-player games, and I really doubt anybody does in 3-player. (...)

I disagree with that statement, since "nobody" and "everybody" are too extreme.

As far as players who are engaging in play with the broader BGA community are concerned, I stand by my statement. No amount of skill can make up for errors by team-mates to that extent.

[Wreckage]

Your rankings of variant difficulty are off a bit. Difficult (5 card variant) is the hardest variant.

It’s not just that you have fewer tokens per game, but the ratio of safe discards to get those tokens is tighter. In avalanche the deck starts with 6 unique cards (the 5s). In “difficult” the deck starts with 9 unique cards (the 5s and m2, m3, and m4). Also as you mentioned, multi 1-4 can bottom deck you.

It not just that your hands are tighter, the game is also shorter. Both games require you to play 30 cards, but in “difficult” you have 5-10 less turns to do it. Fewer tokens, more cards to save, and fewer turns.

In all other variants black powder makes the game harder. But in 5-card it probably makes the game a little easier, allowing 5 more safe discards while only adding 1 more unique card.

Playing both variants with the same team…
If you play 10 games of avalanche, with no flams or bp, you might have 8-10 perfect scores.
If you play 10 games of difficult, with no flams or bp, you might have 2-3 perfect scores.

I guess I made my case between vanilla games. But I think that vanilla difficult is harder than avalanche with bp.