ELO on luck based games

[ChiefPointThief]

My statement isnt that bold. There are roughly 28x more players to ever play azul. In addition (if I remember correctly) there are 11 games of azul played every min on this site. Whereas if you look at sobek 2p there were only 36 games in total in the last 24hrs. Not to mention some of those are from a regular who has me blocked, friend only games, and tb. My point being there are probably more ganes of azul played every min (you can even bump that up to 5 min if you want) than whats available for an entire day of a sobek 2p player.

Whether there are 25,000 or 750,000 players I would still be in the upper percentile of players. And it is like a pyramid. The more elo in the community the more elo there is for the upper percentile of players to feed off of. And while this game has luck it still isn’t as lucky as the “4 & 5” luck rated games. Considering I reached 600 as is. Saying I could reach 700 if this game had a community like azul isnt a stretch at all

[FrankJones]

My statement isnt that bold. There are roughly 28x more players to ever play azul. In addition (if I remember correctly) there are 11 games of azul played every min on this site. Whereas if you look at sobek 2p there were only 36 games in total in the last 24hrs. Not to mention some of those are from a regular who has me blocked, friend only games, and tb. My point being there are probably more ganes of azul played every min (you can even bump that up to 5 min if you want) than whats available for an entire day of a sobek 2p player.

Whether there are 25,000 or 750,000 players I would still be in the upper percentile of players. And it is like a pyramid. The more elo in the community the more elo there is for the upper percentile of players to feed off of. And while this game has luck it still isn’t as lucky as the “4 & 5” luck rated games. Considering I reached 600 as is. Saying I could reach 700 if this game had a community like azul isnt a stretch at all

Fair enough. You and one other player are at expert level of Sobek 2-player, so perhaps 700 would be attainable with more players in the player pool.

The "luck" factor is assigned by the BGA developer. Do you agree with the luck factor of 3 for Sobek 2-player? (I know nothing about the game).

What luck factor would you assign to Pineapple OFCP (as played here on BGA with 95% of games being 6 hands)? What factor(s) would you attribute the very low Elo ceiling of OFCP to?

[dschingis27]

My apologies for the delayed reply. (Earlier in this thread I quoted the above portion and said I had more thoughts that I would post soon).

First of all, thank you very much for writing it up and laying out your thoughts so clearly. I do enjoy these nerdy discussions and I'll try to respond to most of your points.

If 2 players have the exact same results across 100 games against the same opponents, that is literally the definition of "two equally skilled players."

No, absoulutely not! 100 games are way to few to make a definitive statement about skill or Elo for that matter. Every competitive Backgammon or Can't Stop player can tell you that stretches of 100 games are usually not representative, both mathematics and simulations show that there is a high variance in win rate in only 100 games. If player A had 200 wins out of 300 games before and player B had 100 wins out of 300 games before, and then they have equal performance in 100 games, every sane person should assume that player A is much better than player B. For games like Backgammon or Can't Stop, it simply does not make sense to rely only on the last 100 games, particularly when you have more games available.

Why should anyone care that the "historical" Elo gap of 100 is mostly erased? It should be entirely erased. Because, one of two things is the case:
1) That 100 Elo gap was artificial in the first place. Maybe each player was actually a true 650 rating, but one had a small win streak and the other had a small losing streak, putting them at 600 and 700 instead of both at 650.
or
2) The lower rated 600-Elo player has improved and is now equal in true skill and Elo to the 700-Elo player.

It is extremely flawed that you think the above two are the only options. These are possible scenarios of course but you missed an important third one that happens many times in practice. The third one is: A good player has a bad (unlucky) stretch of 100 games and a bad player has a good (lucky) stretch of 100 games, making both have similar realized performance but not similar skill. Again every experienced and Backgammon or Can't Stop player can tell you that this happens many times.

All in all you are simply underestimating variance way too much.

Either way, one purpose of Elo and k-factors is to get players accurately rated, and as quickly possible, right? (Hance the higher temporary k-factors during a player's first 11 and 21 games played). So, the results you showed, in which higher k-factors closed this 100-Elo-gap more quickly - to me, that is not a flaw in k-factor; that is the k-factor working efficiently to correct inaccurate Elos.

If the only consequence of current K factors was that two players with similar performance in the last 100 games have the same Elo, I wouldn't mind. (Particularly if those last 100 games were weighted equally.) I don't think it's accurate but it would be good enough. Unfortunately you missed my main point of my old simulation post. My main point: It is not about what happens after 100 games. It is about what happens after 15 or 30 games! My complaint was about the shape of the curve, not about the endpoint after 100 games. With K=40, the original Elo gap of 100 is halfed already after only 15 games and an Elo gap of only 20 is reached after only 30 games! That's just insane nonsense for games like Backgammon or Can't Stop. 15 or 30 matches are nothing. With K=20, the original Elo gap is halfed after 30 games and is reduced to only 20 after 60 games. This is much better already but still nonsense. The problem really is how strongly the most recent games are weighted for the Elo calculation. The last 10 games are 5 times more important than game 61 to 70 before now. And even within the last 10 games, the last 2 games are much more important than game 9 and 10 before now. This effect is modulated with the K factor, that's what I wanted to show with my simulations.

(The strong weighting of most recent games is plausible for complex games with steep learning curves (Terra Mystica, Through the Ages) but not for Lucky Numbers et al.)

I took a quick glance at lifetime Elo and Arena scores for "Earth" And "Terraforming Mars" For both games, the spread in Elo (from highest to the floor of 100) is similar to the spread in Arena points (from highest to the lowest shown, keeping in mind that the truly lowest arena scores might not be visible because the actual arena score is hidden for any player who is not yet purple elite.)

Spreads are not good indicators if the rating system works good or bad. The spread doesn't tell you if the right people are in the right positions. Plausible spreads also happen with bad rating systems.

So, despite the double k factor of Arena (40) compared to standard games (20), the arena results seem to reasonably mirror the Elo results, despite the fact that Arena scores get reset every 3 months and despite the lower overall player pool.

I will show the difference of K=40 vs. K=20 again in my next post.

One more thought. A lower k-factor could make it take longer for players who should have a wide Elo gap to actually attain that gap. Which means, there is a longer delay in getting players accurately rated, and a higher number of games adversely affected by the fact that one or both players have an inaccurate Elo.

This is a very valid concern. It could be solved by having higher K factors in your first, say, 100 games before we end up with K=10. Let's say after the first 20 games with K=60 / K= 40, the next 30 games are at K=30, the next 50 games are at K=20 and only after 100 games we start with K=10. Today I also think that my original suggestion of K=5 was reaching too low. Maybe having K=10 as lowest possible K factor is good enough, still we could have a system with 3 categories, K=10, K=15, K=20.

In any case, you desire a quick alignment of calculated Elos with "true Elos", however, this is not possible in the first place for Lucky Numbers et al. High K factors make rating accuracy much worse for everyone for whom a lot of data available and it only makes rating accuracy a little bit better for those with small data available.

In either case, over a large enough sample, the results should end up where we expect.

No, absolutely not! This is one of the most common myth that I hear about the Elo system. In reality, Elos do not "converge" (and the less so, the higher the K factor). I will show that in my next post. To me, convergence means that a series of values tends towards a certain value and then stays at that or close to that value. This is intrinsically impossible with the current Elo system. In reality, the Elo values inevitably bounce around in a wide margin. It is true that the range where the Elo value bounces is related to your "true Elo" but when your Elo values bounce around by +/-100 which is usual, you have no real chance to know where your "true Elo" really is. And it is not getting better over time because Elo cares so much about most recent results. Only if the skewed sample with overweight of recent games has some sense (e.g., in Terra Mystica), then your Elo is very meaningful, but this skewed sample is just causing an inescapable Elo bouncing in high-luck games.

And, yes, higher-luck games have higher variance and therefore require more games to meet the criteria of "large enough sample". Some may view this as a problem; I do not.

Here you are contradicting yourself now. If you already agree that larger sample sizes are needed for higher-luck games, then you should favor lower K factors for those games. Only with lower K factors, the "large enough samples" make their way into the Elo calcualation. With current K factors, the realized sample sizes are way too short and heavily skewed towards most recent games. Above you said the quick Elo adjustment with high K factors are not a problem but now you say we need large sample sizes. You can't have both. I will show more on this in my next post.

[dschingis27]

I made another simulation to make my points more clear. Again, I simulated the Elos of two players over the course of 100 games. This time, both have the same "true Elo" and the same starting Elo. I fixed their "true Elo" at 300 and they are facing opponents with Elo 200 in all their games. That means both players have a constant win probability of 64%.

In the pictures below you can observe how their Elo evolves and wobbles over time with the different K factors (I made 2 runs per K factor). The blue line always shows the current Elo difference between the two players.

K=40


K=20


K=10


I made another simulation where I made 1000 runs instead of 2 runs for each K factor. For each K factor, I calculated the relative frequency of Elo difference categories. (So the follwoing numbers are based on 1000 runs of 100 games each.)

Table from 1000 runs for each K factor:



You can see that with K=40, the Elo difference is greater than 50 in more than half the time, it is more than 100 in 21% of time. With K=20, still in one third of the time the difference is more than 50. Only with K=10 we get something that I would consider acceptable.

Why would we want an Elo system that is so inaccurate so many times?

I know that some of you will argue that my assumptions with constant true Elos were unrealistic. But that is more than good enough for the question how accurate the Elos are measured. Even in such a simple situation with constant true Elo and constant opponent strength, we get such strong wobbling and inaccuracy of the calculated Elos. This is the simplest situation a rating system can get. There is no reason to assume that it would get better if you count in player form and learning curves and more variance in opponent strength. Maybe I do a simulation with these factors in the future, too.

[FrankJones]

I think we are going around in circles here, and I think I know why. But, at the moment I am too tired (literally, didn't sleep well last night) to elaborate.

[Edit a few minutes later]

Okay, I'll try.

You are simultaneously saying "Elo here at BGA is inaccurate" and "It's very important to preserve the Elo of any 2 players at one snapshot in time."

I see this as you contradicting yourself. You also indicate that a difference is 100 Elo is huge. Specifically, you said that two players with an Elo gap of 100 means that the higher rated player is "obviously much better" than the lower rated player. How can you say that if the Elos are not accurate in the first place?

If The Elos fluctuate too much (which seems to be your primary complaint), then at any given moment, we have no idea what the actual skill difference is between a 300 Elo player and a 400 Elo player. So, for me, I don't care if those two players have played 100, 1000, or 3000 games; if they play their next 100 games and have identical results, I would say THAT is a more likely indicator of their skill level - which I would interpret to be nearly equal - than their 100-Elo gap prior to those 100 games. And I don't have any issue with their Elo gap disappearing over those 100 games, nor do I have an issue with how fast their Elos converge.

In general, I don't consider it an issue that Elo here at BGA jumps around. I've repeatedly explained why I feel this way. You feel differently, and that's fine. It doesn't mean either one of us right or wrong. To me, fluctuating Elos are par for the course in the world of board games with a luck component.

None of this stuff bothers me. I accept is as is, and to me, the question of whether any of this is a "problem" is entirely a subjective matter of opinion. Even in a game with no hidden information, players can go on winning streaks or losing streaks, and Elos can fluctuate. This could happen in chess, which, as has been noted numerous times, is what Elo was originally designed for.

Using the logic presented in this thread, if a 2100 chess player and a 2000 chess player have a losing streak and winning streak, causing their Elo to temporarily converge, that would indicate a flaw in the Elo system. Yet, the main point I have seen in this thread is something along the lines of, "Elo works for chess but not for games here at BGA if those games have any luck component".

It's all circular reasoning because the reasoning relies on an assumption that is contradicted by the supposed conclusion.

Keep in mind, I'm not some doofus when it comes to statistics and probability. In my 10-person cohort of undergraduates majoring in actuarial math, I was the only one who passed the actuarial exam. And I didn't just pass it; I scored much higher than the required passing score.

[FrankJones]

Why would we want an Elo system that is so inaccurate so many times?

I could counter by asking "why should we care?"

Let's use the game "Earth" as an example. A highly complex game with great skill, but subject to short-term variance because all card draws are random.

A few weeks ago, I and the other top 8-10 players were tightly bunched in the 630-670 Elo range. (This is very rare, but it did happen.) However, the historical graph provides much more useful information.

If you look at the historical graphs of myself, kevinsbane, and mnaveenkumas2009, you can see that those 2 players historically trend between 700 and 850, giving a decent idea of their skill level, whereas I have been flatlining around 670. So, yes, at any one moment in time, the leaderboard might look different, but I can say with reasonable confidence those are the two best Earth players here. And most of the time, you'll find them in the top 2 Elo spots on the leaderboard. And in the Arena leaderboard, despite the doubled K-factor.

So, to me, the system is working.

In OFCP, yeah, I'm (very minimally) mildly disappointed not to have a 500 Elo, since I know for fact I am an expert player. I would love the opportunity to play for real money and I would sit down and play with anyone. But I accept that here at BGA, OFCP is an absurdly high variance game that makes it impossible for ANYONE to achieve a 500 Elo or even close to it. And I don't really care. As I have said, the higher and lower Elo ceilings and Elo spreads or various games give (for me, anyway) a lot of useful information about those games.

I do not see any inherent necessity of having every game have the same Elo ceilings and Elo spreads.

[Andrei3009]

I think we are going around in circles here, and I think I know why. But, at the moment I am too tired (literally, didn't sleep well last night) to elaborate.

[Edit a few minutes later]

If The Elos fluctuate too much (which seems to be your primary complaint), then at any given moment, we have no idea what the actual skill difference is between a 300 Elo player and a 400 Elo player. So, for me, I don't care if those two players have played 100, 1000, or 3000 games; if they play their next 100 games and have identical results, I would say THAT is a more likely indicator of their skill level - which I would interpret to be nearly equal - than their 100-Elo gap prior to those 100 games. And I don't have any issue with their Elo gap disappearing over those 100 games, nor do I have an issue with how fast their Elos converge.

I'm sorry to intrude into your argument. In "circular situations" it is useful to think more deeply about what is at stake. The key question for me is "what is a 300 or 400 elo player"?

For a formal definition, you need to set one value of elo (1200 in chess).All other players will be determined by the winning percentage formula, for example, 1350 is the player who wins player 1200 ~ 74% of the time, and so on. This is what is called "try elo". This definition doesn't work perfectly on bga due to the 100 elo flor, but that's another question.

Without assuming this "saying 300 elo" a player doesn't make sense for me.

[Jellby]

This definition doesn't work perfectly on bga due to the 100 elo flor, but that's another question.

Not only that, but the fact that everyone starts at 0. I believe in chess.com, even if your are a total beginner you start with at least 400. If you start close (or even below) the floor, it will have an effect. If you start well above, it may never have any influence.

The problem (or another problem) is elo is based on absolute differences, but we tend to judge the rankings based on relative differences. For instance, the expected outcome of a match between players with 100 and 200 elo, or between players with 2100 and 2200 elo is exactly the same (according to elo). But we tend to judge that in the first case one player is twice the other (although that's irrelevant), while in the second case they are "almost the same".

Also, according to Wikipedia: "The USCF maintains an absolute rating floor of 100 for all ratings. Thus, no member can have a rating below 100, no matter their performance at USCF-sanctioned events." So the elo floor is not something unique to BGA.

[ChiefPointThief]

Why would we want an Elo system that is so inaccurate so many times?

I could counter by asking "why should we care?"

Because you entered a thread about it. It would be like me going to the tournament tab and going to every thread there and posting "You shouldn't care about this issue. I'm not a tournament player. Go play arena or regular games. The world won't stop spinning if bga doesn't enhance their tournaments".

Tons of ppl DO care about it. I haven't peeked in the suggestion tab in a long time. Whats the first thing I see when I go there today? Another one of these threads. EVERY arena season someone is in these forums complaining about this same issue. There are bug reports asking about this issue dating years ago.

If The Elos fluctuate too much (which seems to be your primary complaint), then at any given moment, we have no idea what the actual skill difference is between a 300 Elo player and a 400 Elo player. So, for me, I don't care if those two players have played 100, 1000, or 3000 games; if they play their next 100 games and have identical results, I would say THAT is a more likely indicator of their skill level - which I would interpret to be nearly equal - than their 100-Elo gap prior to those 100 games. And I don't have any issue with their Elo gap disappearing over those 100 games, nor do I have an issue with how fast their Elos converge.

I think you are still missing his point. The gap is cut in half after only 15 games in aes and 30 games regular not 100. Which is literally nothing for luck based games. To give an example I recently had a bad game stretch in captain flip losing 7 of 10. Going from 408 to 333 which is 17th to 144th all time. The difference in chess is that you aren't going to lose x amount of games based off of luck. I've seen games in which my opponent made suboptimal moves all game and still won this isn't possible in chess.

[tbhp]

There are bug reports asking about this issue dating years ago.

Seems like a weird place to put complaints in since the choice of how Elo should work remains a subjective one.

This raises the question of where can people give feedback on things that are subjective in nature. I suppose the bug feature may get used for this purpose too, but it seems off.