BGA has a serious bully problem

[FrankJones]

The more luck there is in a game, the less value Elo has for predictive purposes. But, the more luck there is in a game, the less skill there is in winning the game anyway, which makes it inherently difficult to predict what will occur anyway.

For a tool to illustrate the results of what has actually occurred, Elo works well for any game. Personally, I find Elo on BGA to be a good indicator of skill in high-luck games or no-luck games. That has been my experience. Others' mileage may vary, of course.

Elo may originally have been created to serve purely as a predictive model, but board game sites have long used Elo for other purposes, mainly, establishing rankings based on what has actually occurred. There is no universal rule such as the gravitational constant that requires people to use Elo for exactly what it was created for decades ago.

[Lucy-mozzi]

You can join me, in your friends, i can playing with you to hanabi. Also i can teaching a new games. Without stress, i know many games. And i understand about what you talk. I feel the same in past but I found a very good friends like playing hanabi without stressed. You can join me and my friends

[BarnardsStar]

Indeed, we would probably do well to bear in mind Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” Elo has ceased to be just a measuring stick for comparison, and has become a goal in and of itself. So one shouldn’t be surprised that it doesn’t bear up entirely gracefully under the added weight.

[tbhp]

3) And on BGA, it works very well. Of the 15-ish games I play, I get similarly skilled opponents when I play people with similar Elo ratings.

Far not well in co-op and high luck factor games. Too easy to level up as beginner and too easy to fall down from expert level.

That’s because in those games there isn’t as much difference between a beginner and a top player. Your problem is with the game, not the system.

You could ask that the k factor be different in high luck games, so that rankings don’t move as easily, but I don’t see the justification why. That would be like hiding the true nature of the game, making the better players appear more in control than they actually are.

[FrankJones]

Agreed, I'm not sure what a lower K-factor would accomplish. First, we would need to somehow determine which k-factor to use for various games.

But, a high-luck game should have more fluctuations and swings than a low-luck game. What purpose does it serve to hide that information behind a lower k-factor? Someone could go on a winning streak and just park atop the leaderboard, and with a low k-factor, people might start complaining, "How can anyone ever catch this person"?

I think exactly that happened in a thread about Euchre, and that's with the existing k-factor.

A question about a no-luck game: Chess:

It's entirely possible for two 2000 Elo players to play and one loses 4 games in a row. Each game is a 50/50 scenario, and as anyone should know, a 50/50 can go in direction 4 times in a row (or more).

If that happened, after 4 games, would people complain that the Elo system is broken, because it predicted the two players would split the 4 games with 2 wins each, and yet one person won all 4?

With a predictive model, the key phrase is "in the long run." Where did this notion come from that the system is broken if someone loses multiple times to a significantly lower rated player? The more important question is, "what would happen if they played 100 times?" And if the +400 Elo player can only win 75% of those games in a large sample, then, well..... that player isn't actually a +400 Elo player. I'm not seeing an issue here.

[Gooorn]

3) And on BGA, it works very well. Of the 15-ish games I play, I get similarly skilled opponents when I play people with similar Elo ratings.

Far not well in co-op and high luck factor games. Too easy to level up as beginner and too easy to fall down from expert level.

That’s because in those games there isn’t as much difference between a beginner and a top player. Your problem is with the game, not the system.

You could ask that the k factor be different in high luck games, so that rankings don’t move as easily, but I don’t see the justification why. That would be like hiding the true nature of the game, making the better players appear more in control than they actually are.

Counterexaple: Texas Hold'em Poker. Luck factor is very high but there IS mutch difference between beginner and top player. In normal ranking system (money) top players looking for play with less experienced players (they called fish) but with present ELO system is better to play with similar or higher ranked player to get higher ranked.

[FrankJones]

Far not well in co-op and high luck factor games. Too easy to level up as beginner and too easy to fall down from expert level.

That’s because in those games there isn’t as much difference between a beginner and a top player. Your problem is with the game, not the system.

You could ask that the k factor be different in high luck games, so that rankings don’t move as easily, but I don’t see the justification why. That would be like hiding the true nature of the game, making the better players appear more in control than they actually are.

Counterexaple: Texas Hold'em Poker. Luck factor is very high but there IS mutch difference between beginner and top player. In normal ranking system (money) top players looking for play with less experienced players (they called fish) but with present ELO system is better to play with similar or higher ranked player to get higher ranked.

True - and that is a combination of factors: Playing for money with the intent to win as much as possible will involve a different path than trying to play competitive games and/or trying to climb an Elo leaderboard.

Poker itself is also an ongoing game not meant to be evaluated in very small sample sizes. The "winner" or "loser" in 6 hands of poker is meaningless. If I fold 6 hands in a row, I'll have fewer chips than at least one other player at the table. That is completely irrelevant to who has played the best poker, who will win the most money in the long-run, etc.

Open-face Chinese poker is exactly that: The "game" (as played here on BGA) is SIX hands of a HIGHLY volatile and high-variance game. An expert player would be happy to win half a point per hand, on average, against an opponent. (Someone could make six-figures a year by sustaining a win rate of 0.5 points per hand at this game).

Yet, dozens of points can swing in a single hand. The long-tern for this game is thousands of hands. I've been running above expectation against a friend on a different app for 4000+ hands. Is Elo well-used here for this game? No, but the issue isn't necessarily the game or the Elo system, but the minuscule sample size of 6 hands after which a game is declared "over".

[tbhp]

Yes, the two goals seem to be quite different. In order to make money, you'd want to prioritize your winrate, not caring about the effect on your elo. Whereas in order to increase your elo, it's not simply a matter of winrate, but of who you hold this winrate against.

Winning 80% of the time against bad players would return you money, but would probably hurt your elo, whereas winning 60% of the time against top players would improve your elo, but wouldn't return as much money.

[ChiefPointThief]

The more luck there is in a game, the less value Elo has for predictive purposes.

So the probabilities shouldn’t be the same in a game w/ a 400v200 elo player w/ zero luck rating as a game with 4 luck rating?

3) And on BGA, it works very well. Of the 15-ish games I play, I get similarly skilled opponents when I play people with similar Elo ratings.

Far not well in co-op and high luck factor games. Too easy to level up as beginner and too easy to fall down from expert level.

That’s because in those games there isn’t as much difference between a beginner and a top player. Your problem is with the game, not the system.

You could ask that the k factor be different in high luck games, so that rankings don’t move as easily, but I don’t see the justification why. That would be like hiding the true nature of the game, making the better players appear more in control than they actually are.

Elo is supposed to be a measure of how skilled a player is in a particular game. What is “elite” in one game may be mediocre in another (like my 68% win rate in captain flip is elite but in a 0 luck game would be mediocre). If you have a system in which a player can play at an elite level for thousands of games and fall several tiers overnight because of a bad streak then that system is flawed.

[MarkSteere]

I’m providing an argument from ChatGPT that is exactly opposite...

I don't see this as "exactly opposite." It's two different topics. One is the predictive value and the other is the ranking value.

You have to consider the bias of the prompt question. If it's a biased question like, "Can you formulate an argument for why Elo is a robust ranking system?" you'll get a biased answer.

But in general, yes, I'm aware that ChatGPT can generate contradictory results based on how questions are put to it. This claim that Elo is a "robust tool" doesn’t make as much sense to me as what it was saying about curve shapes and predictability. That didn't just "appear to make sense." It made sense.

If you expect me to derive a rigorous mathematical proof, that's not gonna happen. It would be beyond my ability, even if I were inclined to invest my time in something like that. But I don't see anyone else doing that either.

Sure, LLMs are "not to be trusted." This is not a profound insight. They're no less trustworthy than laypeople, which appears to be what we have here.

One thing you have to consider is the degree of luck in a game. Take the extreme case of just flipping a coin and trying to get heads. I'm seeing this argument of "Sounds like you have a problem with the game." No. I don't have a problem with the game. I have a problem with attempting to rank players in high luck games. It's not "effective" or "robust" or anything along those lines. It's more like pointless. The more luck there is, the more pointless it is.

Backgammon involves a high degree of skill. But rankings are still vulnerable to luck streaks. Ranking is a course and crude measure of skill. If you're extremely skilled, that will be reflected in the rankings. If you're a little above or below average, luck streaks will obscure small differences in skill, unless you play a million times and it's all included in the evaluation.