ELO on luck based games

[FrankJones]

Why would we want an Elo system that is so inaccurate so many times?

I could counter by asking "why should we care?"

Because you entered a thread about it. It would be like me going to the tournament tab and going to every thread there and posting "You shouldn't care about this issue. I'm not a tournament player. Go play arena or regular games. The world won't stop spinning if bga doesn't enhance their tournaments".

Tons of ppl DO care about it. I haven't peeked in the suggestion tab in a long time. Whats the first thing I see when I go there today? Another one of these threads. EVERY arena season someone is in these forums complaining about this same issue. There are bug reports asking about this issue dating years ago.

I did NOT say "no one should care".

I said,

"I could counter by asking, 'why should we care?' "

Those are not the same thing. Yes, I entered a thread about it. And upon viewing this thread, I read posts that did not present a compelling enough case for me to believe that the supposed flaws exist or are anywhere near as bad as presented in this thread. So, I have presented some counterpoints as part of a calm and reasonable suggestion.

I didn't realize I wasn't allowed to do that.

How exactly do you know what percentage of players for the game(s) in question actually care about this issue or think it's a problem that needs to be addressed? For all we know, it could be 10%, 5%, or even less.

For the record, I DO care about Elo and possible flaws in Elo. I just don't agree with the arguments made in this thread.

[FrankJones]

I think you are still missing his point. The gap is cut in half after only 15 games in aes and 30 games regular not 100. Which is literally nothing for luck based games. To give an example I recently had a bad game stretch in captain flip losing 7 of 10. Going from 408 to 333 which is 17th to 144th all time. The difference in chess is that you aren't going to lose x amount of games based off of luck. I've seen games in which my opponent made suboptimal moves all game and still won this isn't possible in chess.

I'll try saying this one more time, in the clearest and most plain English I can:

I am not missing the point. I just do not agree with the point being made.

And I will continue to point out the obvious circular argument being made:

You are saying Elo in these games is severely flawed. Yet, you are utterly convinced that a player with a 408 Elo should actually have that Elo, an assumption which you then use to prove your argument, when in fact, one of your primary premises is contradicted by your conclusion.

I have raised this point many times and you have not replied to it.

For all we know, the 333 Elo you had after your losing streak is more accurate. Or any number between 408 and 333. The fact that you dropped from 17th to 144th is irrelevant. A single loss in Wingspan could drop a player by a very high number of ranks on the Elo ladder.

If these games are as truly luck-based as you say, I view that as a failure in game design, not a failure in the Elo system. You view it differently. That's fine. You feel as though you have proved your point. I feel as though I have proved mine.

We could just leave it at that and agree to disagree, or we could keep repeating ourselves.

I see little value in the latter.

[dschingis27]

I think we are going around in circles here, and I think I know why. But, at the moment I am too tired (literally, didn't sleep well last night) to elaborate.
[Edit a few minutes later]
Okay, I'll try.

First of all, I have to say that my large response post to you got a bit of an aggressive tone that I don't intend and I am sorry for it. I was a bit in a rush and I can get very passionate about my points, I don't intend to question your intellectual level or anything which was creeping into the post a little bit between the lines. I know that you have an analytical mindset and we agree about many things you wrote in this thread, you also made an excellent summary how Elo works. It's just about the K factor that for some reason, as you said we are going in circles. I will probably write a little more in the next days but I want to respond to your main point and I try to keep it not too long.

You are simultaneously saying "Elo here at BGA is inaccurate" and "It's very important to preserve the Elo of any 2 players at one snapshot in time."
I see this as you contradicting yourself.

I don't think it is contradicting. I think it it is important that the Elo of a player can be roughly preserved because it is the usual case that the skill levels / expected win probabilities of players do not change or at least do not change through short series of games of 10, 20, or 30 matches. The problem with current Elo is that it cannot even handle this usual "base case". You wrote it yourself:

If The Elos fluctuate too much (which seems to be your primary complaint), then at any given moment, we have no idea what the actual skill difference is between a 300 Elo player and a 400 Elo player.

If the Elos fluctuate too much, we don't even have a chance to measure skill difference with them, you said it yourself. Hence the logical consequence to fix this is to make Elos fluctuate less. Fixing the fluctuation is a necessary (not sufficient) condition for accurate skill measurement, it's inevitable to have the situation covered where skill levels don't change much over time.
"It's very important to preserve the Elo of any 2 players at one snapshot in time" That wasn't exactly my point but it was "It's very important to preserve the Elo of any 2 players at one snapshot in time if they have an equal/similar and relatively constant true Elo" This is important because having similar true Elo thoughout series of games is not unrealistic, it is the usual scenario. Constant true Elos troughout 100 games is probably unrealistic but it made it possible to investigate the Elo inaccuracy introduced only by natural Elo fluctuation through randomness, not by any other factors.

It seems to me that you often think of Situation A below while I often thnk of Situation B below, leading us to different conclusions.
Situation A: The Elos at a certain time point are strongly influenced by randomness and likely inaccurate but if 2 players have similar performance from that point onwards, that is likely caused by skill.
Situation B: If 2 players have similar Elos at a certain time point, that is likely caused by skill but what happens from that point onwards is heavily influenced by randomness.
I feel we were often in the above opposite grounds. But the fact of the matter is that both skill and randomness are in both historical and recent/future series of games. And a lower K factor helps with both situations A and B! A lower K factor would make situations - where 2 player's Elos deviate strongly because of mostly luck - much less frequent in the first place. If the calculated Elos rely on longer series of games and are less sensitive on only most recent games, then the main way to cause variance in Elo is actual skill, not luck. Additionally, a lower K factor helps that Elos can be preserved if they happen to be close to their true value already.

You made the point that a higher K factor helps to resovle situations where player's Elos deviate strongly only by randomness because they can then move back to "true Elo" quicker. But the problem is what happens after that. With the current K factors, the Elos will inevetably heavily bounce away from true Elos again. That is what causes overall inaccurate Elos in the long run.

Apart from all the above, I think we can agree that we have to distinguish two general topics:
(1) A lower K factor would make Elos more accurate.
(2) What degree of accuracy do we want in our rating system?

I think that (1) is an irrefutable objective fact. But (2) is a valid subjective question. As you pointed out, you don't need to care about accuracy of Elos and maybe people do enjoy having the fluctuating Elos for some reason (e.g., as a metric of what happened recently). We should just not deceive people and make them think that calculated Elos in the current Elo system are mostly influenced by their skill, they are only very partially influenced by their skill and very heavily influenced by luck. And it is a fact that different K factors do modulate that balance.

Specifically, you said that two players with an Elo gap of 100 means that the higher rated player is "obviously much better" than the lower rated player.

I think you remembered this wrong. I don't think I said that anywhere.

[FrankJones]

I might not be able to respond to everything tonight, but:

1) Regarding your first paragraph - you have nothing to worry about or apologize for.
I didn't perceive any aggressiveness or negativity in your posts. I would describe all your posts as friendly and polite.

2) second paragraph: I was saying "if the Elos fluctuate too much...."
But in my opinion, they don't fluctuate too much. I think it's fine the way it is. We need to keep in mind that "fluctuate too much" is in itself a vague, subjective phrase. What constitutes "too much"?

Most sites I've played on before BGA used a k-factor of 32. So, to me, 20 already seems low. If I lose 12.8 Elo because I lost to a player 100 Elo below me... I don't care, and I don't see this as a problem. Perhaps this is because the current Elo system has not stopped me from achieving strong results in most of the games I play. I can't say for sure. I mostly play "high-skill" games, but they still have a lot of luck involved. (A game can simultaneously be "high-skill" and also involve a lot of short-term luck and variance, in my opinion.)

I've mentioned Pineapple Open face Chinese poker a lot. I'm currently Elo number 2. With a 300 Elo. I'm close to 100% certain I'm better than anyone here, but if I played regularly, my Elo might fluctuate between 220 and 300, perhaps getting as high as 350 if I went on a lucky streak.

I don't care. This does not bother me, and I don't see this as an issue that needs to be "fixed".

If I want to know my realistic level, I would much rather look at my Rolling average Elo over my last X games than my Elo at any one point in time.

Would it be fair for me to suggest that perhaps some people here on BGA (and here in this thread) are likely to perceive their own skill level as being equal to the highest Elo they've ever achieved, rather than some sort of rolling average? To me, what I'm hearing in this thread (not necessarily from you, I cannot keep straight who said what anymore) seems to amount to this:

"I reached [highest personal Elo I have ever had] and then, because there is so much luck in the game, I went on a losing streak, dropped 70 Elo, fell 80 spots on the leaderboard, and now I struggle to get back to [that same highest ever personal Elo]."

What kind of bizarre reasoning is this? Why should anyone look at a historical Elo graph and presume their skill level to be a single point on the graph where the Y-value reaches its apex?

Look at my personal graph for "Race for the Galaxy". What's my likely skill level? The 353 I briefly hit for a few days? Or the 300 that seems to be where I consistently am rated?

I have not played that game much recently (or, at all), but even if it did, I would likely have a few more occasional spikes toward 350, but that flat-line (representing my inactivity in the game) would likely still be there, just with a few occasional peals added in. Those occasional peaks are the outliers. If someone hits 500 in some game with high luck, then drops back down and is mired in the low 400s or high 380s for a while, then gets back to 500 briefly, then repeats, I would view the peaks at 500 as the outliers, and I would look at the mostly horizontal line around 400 Elo.

In short, I'm suggesting that maybe the people complaining that the Elo system is not accurately rating their skill level - maybe those people are not actually at the skill level they think they are.

I'm not sure I agree that a lower k-factor would help to the extent that you imply it will. I mean, taking that argument to its conclusion, we would eventually arrive at a k-factor of 1. There might always be someone complaining that the k-factor needs to be lowered.

Regarding this statement:

"We should just not deceive people and make them think that calculated Elos in the current Elo system are mostly influenced by their skill, they are only very partially influenced by their skill and very heavily influenced by luck. And it is a fact that different K factors do modulate that balance."

I just do not agree at all. Look at my profile! There are some games in which I have close to a 100% winning percentage. Are you saying I should be attributing that to luck and not skill?

What?

Look at my win percentage at Earth, and my Elo. That's luck and not skill ? I find this assertion baffling.

Your final sentence:
My apologies; I must have attributed to you something said by someone else, perhaps ChiefPointThief. I know someone in this thread said that.

[FrankJones]

If 2 players have the exact same results across 100 games against the same opponents, that is literally the definition of "two equally skilled players."

No, absoulutely not! 100 games are way to few to make a definitive statement about skill or Elo for that matter. Every competitive Backgammon or Can't Stop player can tell you that stretches of 100 games are usually not representative, both mathematics and simulations show that there is a high variance in win rate in only 100 games. If player A had 200 wins out of 300 games before and player B had 100 wins out of 300 games before, and then they have equal performance in 100 games, every sane person should assume that player A is much better than player B. For games like Backgammon or Can't Stop, it simply does not make sense to rely only on the last 100 games, particularly when you have more games available.

Hey, I just wanted to apologize again. I feel terrible when I accidentally misquote someone.

You are 100% correct in that I completely misquoted you.

Quoted above is what you actually said, which bears zero resemblance to what I accidentally misquoted you as saying.



Okay, with that out of the way, I'm still not in agreement.

If player A and player B in the above scenario did what you said, I would consider the possibility that player B has studied, learned, improved, watched strategy videos on YouTube, etc.

Maybe player A has regressed. I believe I am currently playing Terraforming Mars at a lower caliber of play than I used to play in the past. For various reasons.

And I certainly would not go so far as the dramatic statement of,
"every sane person should assume that player A is much better than player B"
(emphasis added to the original quote,)

Furthermore, how do we know the opponents' skill levels? Maybe player A was playing weak players, thus winning 200 out of 300, and player B was playing strong players and thus won 100 out of 300.

In Race for the Galaxy I have a sub-.500 win percentage yet I have a 300+ Elo. Why? Because I seem to be stuck in the very low 300s, so when I set my filter to "strong players", Most of my games are against players higher rated than I am. Which means I can play below .500 and maintain my 300 Elo. I'm not expected to win 50% of the time if I am 304 Elo and all my games are against players in the 300-500 range. If I go on a streak, I get up to 350, but then, on average, my opponents are equal to my Elo, and then my 48.6% win rate causes my Elo to drop back down. And, that's fine. I could complain about the system, or I could just ... start playing better. I don't study the game. I do not watch replays. I do not read strategy posts or watch strategy videos. No wonder I am stuck at 300. But if I spent the next month studying the game, does anyone think I am incapable of reaching 500? Based on my stats in my profile for the 15 games I play?

Earth: I have been grinding and grinding toward the elusive 700. I'm a consistent top 5 player in Arena and overall Elo. I just took a devastating loss. Heartbreaking. Soul Crushing. The most stunning end-game sequence I have seen. I thought my opponent's probability of winning going into the final turn was 0. Literally 0.0%. I did not think it was mathematically possible. There was ONE card in a 370-card deck that I thought, MAYBE that card could so it, but I still don't think so.

Of course, my opponent planted that card. In a deck of 370 cards, he had that one card. And then someone found a way to score another 10 points on top of the 33 points that card scored, Because I had myself ahead by 37 or 38, I think. I still need to watch the replay from my opponent's perspective to see what happened.

It took me a long time to get from the mid 650s to 682. And this loss hit me with minus 17 Elo (and minus 32 Arena points, but I don't recally care about that).

I very strongly believe I am a top-5 Earth player on this site. I strongly believe I am capable of achieving and then maintaining a 700 Elo. But, this brutal loss knocked me back to 665. An emotionally crushing defeat. It might take more 15 more wins to get back to 682, depending on whom I play.

Okay, so what if we had a lower k-factor? With a k-factor of 10, I never would have been anywhere near 700 in the first place. How and why would that improve the situation? With a k-factor of 10, it would take 300, 400, maybe 500 games to reach 700. Maybe more. That would take years. Maybe some people do not want to have to grind away for years just to reach 700.

After the game, my opponent (whom I play often; we just get matched a lot in the arena) and I discussed for at least 5 minutes the crazy insanity of the improbable and dramatic finish. (I lost by one point in a game that has scores in the 180-270 range.)

I could complin about how unlucky I got. I could complain about the Elo system. Or, I Could tip my hat to my opponent (which I did, congratulating him multiple times on an amazing and awesome win), and acknowledge that I just simply should have played better and won the game. With a score of 214-213, I surely could have done something different. The game has a very high number of decisions to be made. I was the player with a 350 Elo gap over my opponent. If I want 700, I need to win that game, not lose by one point. .

Now I'm back at 665 and highly discouraged. But I am not annoyed at the Elo system or bad luck. I'm annoyed at myself for not winning the game.

I am not going to write a post and start a 12-page thread called "Look how unlucky I was and how unfair this is and how broken he ELo system is."

My opponent had played me a lot recently. Maybe he has reached a turning point. Sometimes things just "click" for a player. I myself was stuck at one point. What did I do? I started paying very close attention to how my strongest opponents played. And, after a game against the top dog, we started chatting. We've been chatting daily since then, mostly about the game. I made a significant change to the way I played, and almost immediately I started an impressive ascension into the mid 600s. Maybe my opponent had been just playing casually, and then woke up today and said, "You know what? I'm going to review all my previous games against FrankJones to see why I've lost to him every time." And maybe while doing so, he realized, "Ohhhh. I get it now." It certainly felt as though I was playing against a player close to me skill level rather than a player 350 Elo lower than I am.

And if this continued for another 100 games, I would absolutely say, "This player has now improved to the point of being my equal at this game". If he and I play another 100 games and all of them are won or lost by one point, yes, I will place almost 100% of my skill evacuations on those 100 games, and I wouldn't care if we had previously played 200,300, or 1000 games.

Anyway, back to my own epiphany and turning point at which I stopped treading water in mediocrity and rose from the low 500s to the mid 650s: So, yes, in that case, I believe my more recent games did in fact give a more true representation of my ability and skill than my previous larger sample. I got better. I studied. I learned. I chatted with the top player on the site. I reviewed game after game. move by move. Etc.

So, I will regroup, recover, study the game carefully, and see what went wrong. Because had I scored TWO MORE POINTS in that game, I would be at 685 instead of 665.

[ChiefPointThief]

Frank, I’ll get into some other stuff you wrote at a later time but I told you already that it is extremely faulty for you to keep using non luck games in your examples.

Also are we completely disregarding the arena aspect in all of this? It isn’t just about where a skill supposedly is by the end of the competition (or at least no real competition should function in that way). It is supposed to be about who performed the best overall throughout the entirety.

You mentioned something about how do we know my true rating shouldnt be 333. Or that I’m actually as good as I think I am being at 400 etc. Well because I reached over 500 numerous times. Ive maintained these high levels over a duration of hundreds of games. & have a win % that only like 0.001 of players have in this game. So yes. My play has proven that I am above the players in the bracket that I’m currently rated at. And it only took going 30% in just a 10 game span to get down to that bracket. After playing thousands of games at tiers above that bracket. It isn’t a great way to measure ratings for such games.

[tbhp]

Also are we completely disregarding the arena aspect in all of this? It isn’t just about where a skill supposedly is by the end of the competition (or at least no real competition should function in that way). It is supposed to be about who performed the best overall throughout the entirety.

I don't think that's quite correct. I think the arena is supposed to be a mode where last minute runs can make a difference. So, player A may have an overall winrate slightly higher than player B (whether that's against other good players, against bad players, whatever) and player B may still win the season if they perform better in the final stages of the season.

I think this is less of a flaw and more something that was intended by the people who designed arena.

So, it's not quite about overall performance, and it's not quite about where a skill is by the end of the competition. It's a little bit of a savant mix of both.

[FrankJones]

Frank, I’ll get into some other stuff you wrote at a later time but I told you already that it is extremely faulty for you to keep using non luck games in your examples.

I play very few non-luck games. Most of my examples have been games that have luck.

Also are we completely disregarding the arena aspect in all of this? It isn’t just about where a skill supposedly is by the end of the competition (or at least no real competition should function in that way). It is supposed to be about who performed the best overall throughout the entirety.

I'm not ignoring it.

That's just how things work. Do you follow professional sports? Who wins the super bowl? The 18-1 New England Patriots? Or the 14-6 NY Giants? I mean, professional sports, with billions of dollars involved, play a long, grinding, six-month season, suffering injuries and travel fatigue, and the trophy comes down to... whoever wins the final game. Or the final best-of-7.

Not everything needs "fixing". Not everything that you think is a problem is actually a problem.

Can you show me any actual proof that more than 5% of players in any specific game have a major issue with the Elo system? I mean, we have a 12-page thread here and I see basically ... TWO people actually complaining about it.

You mentioned something about how do we know my true rating shouldnt be 333. Or that I’m actually as good as I think I am being at 400 etc. Well because I reached over 500 numerous times. Ive maintained these high levels over a duration of hundreds of games. & have a win % that only like 0.001 of players have in this game. So yes. My play has proven that I am above the players in the bracket that I’m currently rated at. And it only took going 30% in just a 10 game span to get down to that bracket. After playing thousands of games at tiers above that bracket. It isn’t a great way to measure ratings for such games.

In your opinion. If you've reached 500 numerous times, then you'll do it again. Remind me again, which game are we talking about here? (I's difficult for me to keep track of everything in this long thread).

And anyone looking at your graph would be able to see that you have consistently reached 500 Elo multiple times. Seems to me the issue isn't Elo, but rather, the way the leaderboard is shown. If the leaderboard did not show Elo at all, but rather, "highest Elo ever reached", you would be in the top spot, and you would be happy, right ? Or, if it showed a rolling average over last x game, you'd be in the top spot, and you'd be happy, right? This whole post seems to be about your annoyance at not seeing your name in lights at the top of the leaderboard.

I believe I am the best Aquatica player on this site. But I'm not even top 10 on the leaderboard. Currently, 11th. I don't really care, and I certainly don't expect anyone else to. Likewise for OFCP.

You care. I get it. Doesn't mean the rest of us should.

Just so we're clear - you're claiming that 999 out pf 1000 players at this game have a lower win percentage than you do?

No offense, but this sounds like a dumb game. If there is that much luck involved, I think it's just a poorly designed game. Just my opinion. No offense to you or the game designer.

[FrankJones]

Are we talking about Captain Flip? Because if you are saying your historical graph shows that you have been a consistent 500 Elo player, we must be looking at two different graphs.

Your Sobek graph shows you to be a consistent 500 Elo player, and there you are at 493 and rated 2nd overall.

[ChiefPointThief]

Also are we completely disregarding the arena aspect in all of this? It isn’t just about where a skill supposedly is by the end of the competition (or at least no real competition should function in that way). It is supposed to be about who performed the best overall throughout the entirety.

I'm not ignoring it.

That's just how things work. Do you follow professional sports? Who wins the super bowl? The 18-1 New England Patriots? Or the 14-6 NY Giants? I mean, professional sports, with billions of dollars involved, play a long, grinding, six-month season, suffering injuries and travel fatigue, and the trophy comes down to... whoever wins the final game. Or the final best-of-7.

This is very misleading because the NFL has two phases (arena has 1) in which you combined both for your statistic. The regular season (qualifier) and the playoffs. The patriots went 16-0 in the season and guess what? The were ranked number ONE as they were supposed to. The reward for finishing number one was that they got home field advantage throughout the playoffs (which is huge) and a bye week meaning they had to play one less game to reach the championship game (which is also huge). In the playoffs the Giants outperformed the patriots going 4-0 while the patriots were only 2-1 (and lost the superbowl). So the Giants outperformed in stage b (playoffs) of the year.

Speaking of a 7 game series if team a wins the 1st two games (which is likely because home teams when more than not) and team b wins the next two games when they go home it is tied 2-2. But in faulty arena team b would rank ahead of stage a. So just two players playing a rematch at the start of the season and the rankings are already technically off after two games.

Can you show me any actual proof that more than 5% of players in any specific game have a major issue with the Elo system? I mean, we have a 12-page thread here and I see basically ... TWO people actually complaining about it.

This is a VERY flawed counter argument and once again one that could be used for EVERY issue raised in the forums. I don't think even 5% of players in total visit the forums so saying that if theres no documented proof of at least 5% of players in a game mentioning an issue that means it is not a valid issue is silly. Furthermore, the biggest thread I've ever seen in the forums is 100pgs long has several millions of views and guess what the topic was? Issues with the elo system

Just so we're clear - you're claiming that 999 out pf 1000 players at this game have a lower win percentage than you do?

Yes. Unless there's 100 players with a higher winrate

This whole post seems to be about your annoyance at not seeing your name in lights at the top of the leaderboard.

Face palm. I think I've been very clear with what my annoyance is. Also if player a outperforms player b by a large margin (especially in bga's world famous "competitive format") but still ranks lower at the end of the season then yes player a SHOULD be annoyed.