ELO on luck based games

[Suoivax]

"Why do you continue discussing?"

----The guy continuing to discuss.

[Ceaseless]

How would you determine the k-factor for different games? I see the rough idea of wanting a lower k factor for lower luck games, but how do you determine luck and work out what an appropriate k factor is?

Just for clarity - they want a lower k-factor for higher luck games.

Whoops, I typed the wrong word there. Fixed.

[FrankJones]

"Why do you continue discussing?"

----The guy continuing to discuss.

Are you being serious?

I was responding directly to his statement that he found the discussion annoying and tiring. I never said I found the discussion annoying or tiring, so why would I not continue to discuss? I was asking why he is continuing to discuss something he finds annoying and tiring.

You took my statement completely out of context, completely changing the meaning of what I said, seemingly in an effort to make me look silly.
Actually, you completely misquoted me. Could you fix that please? I did not say the sentence you put quotes around.

Personally, I find the discussion interesting. Hence my continued participation in the discussion.

[tbhp]

How would you determine the k-factor for different games? I see the rough idea of wanting a lower k factor for higher luck games, but how do you determine luck and work out what an appropriate k factor is?

Yeah, that seems like an instance of putting the cart before the horse to me. Games shouldn't be classified based on some human intuition of how luck based they may be.

The problem with luck is that you can never know anything for sure. Sure, if a game is a literal roll of dice, then you can say that it is totally luck based, with even odds across all players. But for the more complex games out there, it's impossible to "prove" how luck based a game is in comparison to another.

Keeping the k-factor the same across all games has the advantage of treating all games equally. It doesn't involve any guesswork on our part, it simply states the rules of the competition and then enters all games into said competition. Any differences of experience you might notice then result from the specifics of each game, which kinda makes sense since, well, they are different games. When I play a luck based game, I want to feel like luck plays a role. I don't want the competition to be artificially extended in order to ensure that the best player is more likely to win it.

This is also how sports work, for the most part. In the Olympic Games, do group stages have different sizes based on the luck factor of a game? If we followed the different k-factors logic, then a sport like football should include more group stage games than a sport like rugby, because the former is much more uncertain in its results than the latter. But that's not what we do. All sports follow the same format because they are part of the same competition. This means that the eventual winner in a sport like football may be more uncertain than in some other sports, and no one sees a problem with that. The higher presence of luck is seen as part of the identity of the sport.

[Fuchur]

Thank you for your simulations. You made it perfectly clear why one NEEDS a low k-factor in high variance games (under the assumption that the player skill evolves slowly enough).

There is another point about ELO which isn't clear : Does it work ? Can a single number (or rather difference of numbers) give a good prediction of who will win (let's restrict to 2 player games for simplification). I've seen studies for chess-ELO which provide evidence that ELO works for chess (sample size was rather small for high ELO-gaps).

Does anyone know of studies for other games ?

I made another simulation to make my points more clear. Again, I simulated the Elos of two players over the course of 100 games. This time, both have the same "true Elo" and the same starting Elo. I fixed their "true Elo" at 300 and they are facing opponents with Elo 200 in all their games. That means both players have a constant win probability of 64%.

In the pictures below you can observe how their Elo evolves and wobbles over time with the different K factors (I made 2 runs per K factor). The blue line always shows the current Elo difference between the two players.

K=40


K=20


K=10


I made another simulation where I made 1000 runs instead of 2 runs for each K factor. For each K factor, I calculated the relative frequency of Elo difference categories. (So the follwoing numbers are based on 1000 runs of 100 games each.)

Table from 1000 runs for each K factor:



You can see that with K=40, the Elo difference is greater than 50 in more than half the time, it is more than 100 in 21% of time. With K=20, still in one third of the time the difference is more than 50. Only with K=10 we get something that I would consider acceptable.

Why would we want an Elo system that is so inaccurate so many times?

I know that some of you will argue that my assumptions with constant true Elos were unrealistic. But that is more than good enough for the question how accurate the Elos are measured. Even in such a simple situation with constant true Elo and constant opponent strength, we get such strong wobbling and inaccuracy of the calculated Elos. This is the simplest situation a rating system can get. There is no reason to assume that it would get better if you count in player form and learning curves and more variance in opponent strength. Maybe I do a simulation with these factors in the future, too.

[FrankJones]

There is another point about ELO which isn't clear : Does it work ? Can a single number (or rather difference of numbers) give a good prediction of who will win (let's restrict to 2 player games for simplification). I've seen studies for chess-ELO which provide evidence that ELO works for chess (sample size was rather small for high ELO-gaps).

The question of whether Elo works would probably be answered:

1) "No", by ChiefPointThief

2) "Yes, but with different k-factors for games with different amounts of luck in the games" by dschingis27

and

3) "That depends on what we are trying to accomplish with Elo", by me.

Elo (whether by intention or not) serves several functions. Maybe here on BGA, Elo serves purposes that make the site owners and developers happy, even if these purposes are different from the stated intentions of Elo (including any stated intentions for the purpose of Elo on this website.)

[Ceaseless]

Thank you for your simulations. You made it perfectly clear why one NEEDS a low k-factor in high variance games (under the assumption that the player skill evolves slowly enough).

And that assumption is not always a trivial one. When you throw in a large enough number of games, player quality changes become a real concern. From personal experience in various games, I often have breakthroughs. Even after plateauing for a while, I can still find myself once again making improvements. These things are not static.

[tbhp]

The problem with arena is that you have to think about how changing the k-factor would affect the playerbase. I mean, climbing in arena is already a grind enough as it is, imagine if now you had to play twice more games to get to your supposed ranking.

Sure, there may be bigger setbacks with the current k-factor at 40, but this same variance serves to give good players a constant hope that with some good streaks they may reach the top spots. I think with a lower k-factor, most people would get discouraged and it wouldn't be good for the site.

As it is, most people seem to be happy with the concept of arena being this format where good streaks can make a difference. If we wanted a totally symmetric system, then we would organize a giant round robin between all participants, but I don't think most people would consider that to be a very exciting format for something like arena.

And it's not like the format gives us absurd winners or anything. The winners of the arena seasons are always among the best players of the game (at least in the games I play). It's just that there is some suspense in who is going to manage to detach themselves from the rest, which I consider a good thing.

[dschingis27]

Why do you feel compelled to continue the discussion with me?

I did that to achieve clarity. Partially tried to convince you but mostly I did that for my own closure. I wanted to lay out the weaknesses in the argumentation clearly for everyone to see.

I feel as though you're telling me my opinion is "wrong", which is an odd approach to take.

I feel that your opinion is wrong because I think it is inconsistent. On the one hand, you say we should look at rolling averages, on the other hand you say that a lower K factor doesn't help. But both would do a very similar thing. I absolutely don't get what you get out of keeping a higher fluctuating Elo in addition to a rolling-average Elo.

And the best way to convince them would be to show them suggestion reports in a large number of bug/suggestion forums for specific games.

That is true. There is one that I can mostly agree with. https://boardgamearena.com/bug?id=14455
There is also this one going into a similar direction I argue for, but I disagree with this one, I think it would make rankings too stable. https://boardgamearena.com/bug?id=30059 I also have low hope that they change the system anytime soon so I wasn't motivated yet to make a suggestion about my K=10, K=15, K=20 idea but I should probably do that.

The question of whether Elo works would probably be answered:
3) "That depends on what we are trying to accomplish with Elo", by me.
Elo (whether by intention or not) serves several functions.

But what are these other purposes? What are you trying to accomplish with Elo? I really can't see it in your posts honestly.
What is the purpose of having completely random spikes in your Elo rating?

For Arena, yes, I can understand that some people prefer the more fluctuating version of Elo, I absolutely disagree that that version is better for Arena than one with lower K in high-luck games, but I can at least understand the point maybe making the competition more "exciting". But for normal Elo that is also used for matchmaking, I don't see any good reason to have Elo deliberately much less accurate than it could be and I don't understand how people can prefer the Elo fluctuating so severely like it is now. And if you prefer a fluctuating Elo, why do you only want it for high-luck games? If you prefer a fluctuating Elo for instance for "excitement", then you should argue for higher K factors in low-luck games to have the same amount of fluctuation there.

The problem with arena is that you have to think about how changing the k-factor would affect the playerbase. I mean, climbing in arena is already a grind enough as it is, imagine if now you had to play twice more games to get to your supposed ranking.

That is not what would happen. Even with K=10, in the high-luck games that are usually short, you can get to your supposed region in the ranking in very reasonable time. The problem is that you then don't stay there. The grinding usually comes from the huge variance you face after you reached roughly the position in the ranking that you should have. I want to quote euklid14 from the other thread.

A lower K-factor would help because it would decrease those big swing and earlier games would influence AES more - not only the last 10 games...

That being said, I have already taken Jellbys advice and accepted the way it is.
In Backgammon I can reach top20 quite easily. Then I can decide to play on and reach top5 or fall back to top500 in the next 10 games or I stop at my top20 spot.
Position 1, 20, or 200 may be the result of the same Arena season with same win percentages.

I think with lower K there would be less variance and hence less misplacement and hence less appeal for grinding. I now see that you commented that, too.

I think with a lower k-factor, most people would get discouraged and it wouldn't be good for the site.

So is grinding a good thing or a bad thing now? You have to decide. In your first paragraph you said that lower K would lead to more grinding which is bad. In your second paragraph you said that lower K would lead to less playing (which is essentially less grinding) which is bad.

My take is that yes, a lower K would incentive playing less games in the short-term, less games per single season. But because the Arena score would be more meaningful and people could care more about it, people would play more Arena seasons in the long run.

Also, in my opinion, exactly if we wanted to reduce grinding, we would need to adjust K factors by game, probably not only by luck factor but also by game length. Because we have the same K factor for Terra Mystica and Can't Stop, you have to play the same minimum number of games in Terra Mystica and Can't Stop to reach a certain EAS level. But one game of Terra Mystica is 10 times as long as Can't Stop. (Of course you have to play more games in Can't Stop because of the luck element but that usually doesn't outweigh the differences in game length.) I wouldn't mind if the very long games would even have a K factor of 50.

I quote sprockitz here from the bug report I linked above:
"Some games take 2 minutes, others take 2 hours. Having the ratings move at the same rate per game means you can achieve a balanced rating in a few hours for some games (i.e. Backgammon, Can’t Stop, Reversi, 7 Wonders, Lost Cities), while for long games (Terra Mystica, Clans of Caledonia) it will take more than 100 hours. While longer games should take longer to reach a ‘true’ rating adjusting the K factor can be used to slightly mitigate the issue."

[Cappa]

There is no money involved, nothing to lose or gain but ELO and maybe pride? Its not a big deal. The system isnt perfect and it would be near impossible to adapt the ELO for each individual game.

Dont worry about ELO and just enjoy the game.

Face palm. Always one of these comments. Also it isnt impossible to adjust the win probabilities to consider luck factor. Actually it is rather simple.

Just don't worry about Elo for luck-based games.

It's less complicated that way. Plus, we'll never all agree as to which games count as luck-based and to what extent.

Don't worry about site crashes, Don't worry about bugs, Don't worry about new games being added etc etc etc. You can apply this to literally every topic.

Actually in another luck based related thread luck factor ratings were pulled from games on another site that lets the community rate games. Those ratings were more on point than bga ratings which are set by the dev. Same thing could be done here.

Avoid playing Arena, because in Arena, you cannot filter by Elo.
Play non-arena games and set an Elo filter to avoid being matched against players with very low Elo, because in a high luck game, you don't want to be playing against players rated lower than you are.

This doesn't solve the problem at all. We've been through this Frank! (Although this tip can be helpful)

Elo is reflecting win probability. If there is a high luck factor, there is not much difference in win probabilities between players, hence players tend to have similar Elos.

Except the win probabilities are inaccurate for high luck games. Creating false elos. Thus it isnt even properly reflecting win probabilities. Which brings players closer to the same level than they should be and results in more skilled players inevitably falling below weaker players because the math is off.

I’d think the answer to that would be to hide the actual Elo values. Its primary rationale, on a site like this, is simply to match people up with players of their level.

Its been stated multiple times in this thread that the purpose of elo is for matchmaking which is not true for this site. Elo is used to determine prestige on this site. Your profile is ranked by xp and where you rank all time on the xp list is posted on the home page. The majority of xp is gained from placement on the all time elo list and arena seasons. So if the gaming site is going to crown players based on these things then it should actually attempt to enhance its rating system which has clear flaws.

Ok bud