ELO on luck based games

[FrankJones]

I’ll write a proof by contradiction:

Assumption: A player’s Elo at any given moment perfectly reflects that player’s true skill level.

Premise 1: A player can, without improving his skill level or learning anything more or new about a game, go on a 4-game winning streak against players with similar Elos.

Fact: It is possible for this 4-game streak to occur in one day or a matter of hours.

Fact: If said winning streak happens, this player’s Elo will be 40 points higher than it was before the streak.

Conclusion: In that situation, we would have a player who was rated x, and then hours or maybe a day later, was rated x+40.

This conclusion contradicts the assumption, because a player cannot simultaneously be an x Elo player and an x+40 player.

Therefore, because the logically and validly reached conclusion contradicts our assumption, we reject the assumption.

End of proof.

The only other way to deal with this would be to reject my “premise 1” rather than rejecting my initial assumption. I find it difficult to attack “premise 1” in that argument.

[Fuchur]

Some people are good in skiing, others in shooting. Lets assume that one can associate at every person a one-dimensional parameter, one for skiing, the other shooting, reflecting their skill perfectly. So a person, say Anne, with skills (100,200) will be better in shooting than Bob with skills (200,100) but worse in skiing.

What outcome should one expect for competition in biathlon ? Is there another -- 1-dimensional -- measure of their respective skills in this sport or does one have to deal with a 2-dimensional skill-meter ? In the later case Anne and Ben aren't comparable.

Coming back to "normal" games : I believe that being good at any game here on BGA is a non-trivial task -- i'm just wondering why one can assume that the skills necessary to be good can be reduced to a one-dimensional parameter.

If you think of a sport like biathlon you might think that -- in that case -- a (at least) two-dimensional skill-measure would be needed, one to measure the skill of skiing, the other of shooting and, without strong conditions on all competitions of biathlon, it shouldn't be possible to reduce the skill-measurement of that sport to one dimension.

I do not understand this analogy.

Here at BGA, each game has an Elo system that, although using the same formula, works differently for each game, separately measuring the results of each individual game.

[Fuchur]

If you think of a sport like biathlon you might think that -- in that case -- a (at least) two-dimensional skill-measure would be needed, one to measure the skill of skiing, the other of shooting and, without strong conditions on all competitions of biathlon, it shouldn't be possible to reduce the skill-measurement of that sport to one dimension.

By “strong conditions” you mean like always running it on the same course, always running on sunny days, that sort of thing?

Yes, something like a fixed relation between the difficulty of shooting and skiing.

And yet… in a lot of team sports it has become fashionable to rate players by WAR—wins over replacement. That is, if that player left the team and you had to replace them with a base-skilled player, a rookie or some such, how many fewer games would you expect to win in a season because you didn’t have that player?

This is, of course, a somewhat ridiculous exercise. You can’t possibly know that. And yet people do it as an exercise in reducing all of a player’s skill to one number. As I say, ridiculous, and yet, it has value. You can disgree over how much of a factor, say, batting average should play in WARs, and several definitions exist, but it does have value in being able to compare a top-notch slugger with a knuckleball pitcher. (Insert apt football [soccer] and cricket analogies here.)

Interesting -- i'm wondering how these calculations work.

Likewise, it may be slightly ridiculous to expect all players to have an Elo and all possible combinations of matchups to obey those Elos. (Classic example, three rock-paper-scissors players, each of whom favor one of the three outcomes.) This doesn’t mean it doesn’t have value, or that Elo doesn’t exist. It means it’s an approximation of something that may not be completely linear. But approximations still have value.

Yes, these 3 players can't be compared with a 1-dimensional number representing their skill. (But some-one with a favorite outcome won't get far in any competition of rock-paper-scissors)

Thus my question : Does anyone know of any validation of ELO-like formulas for games (except chess) ...

[FrankJones]

Some people are good in skiing, others in shooting. Lets assume that one can associate at every person a one-dimensional parameter, one for skiing, the other shooting, reflecting their skill perfectly. So a person, say Anne, with skills (100,200) will be better in shooting than Bob with skills (200,100) but worse in skiing.

What outcome should one expect for competition in biathlon ? Is there another -- 1-dimensional -- measure of their respective skills in this sport or does one have to deal with a 2-dimensional skill-meter ? In the later case Anne and Ben aren't comparable.

Coming back to "normal" games : I believe that being good at any game here on BGA is a non-trivial task -- i'm just wondering why one can assume that the skills necessary to be good can be reduced to a one-dimensional parameter.

Thank you for the clarification; I understand your example now. But I don't see the analogy to BGA games.

Yes, saying someone is "300" at "biathalon" seems a bit off. Here at BGA, each game has its own rating per player. We don't take "Earth" (high skill) and "Pineapple Open face Chinese poker" and give a player a single combined rating.

If you want to see different individual components within a game, okay? The end-game stats show how each player did in each category; but, so what? Winning or losing one category is irrelevant. The final score is what matters, and it's either a win or a loss.

[FrankJones]

Thus my question : Does anyone know of any validation of ELO-like formulas for games (except chess) ...

Go? Othello? GoMoku?

I'm, confused. Is there a belief that Elo is not valid for those games? What exactly would such a "validation", as you phrased it, look like?

[BarnardsStar]

Thank you for the clarification; I understand your example now. But I don't see the analogy to BGA games.

Yes, saying someone is "300" at "biathalon" seems a bit off. Here at BGA, each game has its own rating per player. We don't take "Earth" (high skill) and "Pineapple Open face Chinese poker" and give a player a single combined rating.

If you want to see different individual components within a game, okay? The end-game stats show how each player did in each category; but, so what? Winning or losing one category is irrelevant. The final score is what matters, and it's either a win or a loss.

No, we don’t mix Earth and Pineapple OFCP… but we do mix TfM (preludes, no corp era) with TfM (corp era, no preludes) with TfM (Colonies + Corp Era + Preludes) etc, and I definitely think, based on the discussions in that forum, if you were to restrict it to one format, Elos would shake out differently than if you restricted it to another format. Or the debates overs whether 2p or 2+p is better… Arena tries to tease that out a little bit, but basic Elo mixes them all together.

[FrankJones]

Thank you for the clarification; I understand your example now. But I don't see the analogy to BGA games.

Yes, saying someone is "300" at "biathalon" seems a bit off. Here at BGA, each game has its own rating per player. We don't take "Earth" (high skill) and "Pineapple Open face Chinese poker" and give a player a single combined rating.

If you want to see different individual components within a game, okay? The end-game stats show how each player did in each category; but, so what? Winning or losing one category is irrelevant. The final score is what matters, and it's either a win or a loss.

No, we don’t mix Earth and Pineapple OFCP… but we do mix TfM (preludes, no corp era) with TfM (corp era, no preludes) with TfM (Colonies + Corp Era + Preludes) etc, and I definitely think, based on the discussions in that forum, if you were to restrict it to one format, Elos would shake out differently than if you restricted it to another format. Or the debates overs whether 2p or 2+p is better… Arena tries to tease that out a little bit, but basic Elo mixes them all together.

Good point, And I don't disagree.

That said, is it wise to have different leaderboards and Elo for the (possibly dozens ) of different combinations of settings for one game?

[Ceaseless]

Some people are good in skiing, others in shooting. Lets assume that one can associate at every person a one-dimensional parameter, one for skiing, the other shooting, reflecting their skill perfectly. So a person, say Anne, with skills (100,200) will be better in shooting than Bob with skills (200,100) but worse in skiing.

What outcome should one expect for competition in biathlon ? Is there another -- 1-dimensional -- measure of their respective skills in this sport or does one have to deal with a 2-dimensional skill-meter ? In the later case Anne and Ben aren't comparable.

Well, how important are skiing and shooting for Biathlon? This is determined by the rules of Biathlon, the way the players approach it, etc. The game serves as the multi-dimensional means of measuring the combination of the various skills involved. From there people pick their methods. I don't know what it's set to for Biathlon or how accurate it is.

[Ceaseless]

Thank you for the clarification; I understand your example now. But I don't see the analogy to BGA games.

Yes, saying someone is "300" at "biathalon" seems a bit off. Here at BGA, each game has its own rating per player. We don't take "Earth" (high skill) and "Pineapple Open face Chinese poker" and give a player a single combined rating.

If you want to see different individual components within a game, okay? The end-game stats show how each player did in each category; but, so what? Winning or losing one category is irrelevant. The final score is what matters, and it's either a win or a loss.

No, we don’t mix Earth and Pineapple OFCP… but we do mix TfM (preludes, no corp era) with TfM (corp era, no preludes) with TfM (Colonies + Corp Era + Preludes) etc, and I definitely think, based on the discussions in that forum, if you were to restrict it to one format, Elos would shake out differently than if you restricted it to another format. Or the debates overs whether 2p or 2+p is better… Arena tries to tease that out a little bit, but basic Elo mixes them all together.

That's certainly a questionable quirk of BGA, which I'd guess has more to do with effort than accuracy. It doesn't really relate to the rating method as much as it does how they grouped the games it got measured for.

[BarnardsStar]

Good point, And I don't disagree.

That said, is it wise to have different leaderboards and Elo for the (possibly dozens ) of different combinations of settings for one game?

Not even a little bit. I don’t think anybody is actually proposing that; I’m definitely not. Just that I think Fuchur is making some interesting points… but I still think that Elo is a very useful approximation just as WAR is for sports.

Personally I’d rather see a system like Glicko2 in place, which takes into account factors like performance consistency (or lack thereof) as well as how long it’s been since last you played (not a decay mechanism, but rather treating one’s rating as being less reliable if you haven’t played in a long time)… but I might as well ask for a pony while I’m at it.