Here are some plots that prove my point (and illustrate what was nicely explained by euklid314).
These are based on a simulation. I compare two players, one starts with Arena score 1700, the other with 1800. (It doesn't matter where exactly I set the scores, what matters is that the difference is 100 at the beginning.) I simulate 100 games of these players against random opponents with Arena scores in 1500 to 2000. In the simulation, both players play the exact same sequence of opponents and get the exact same sequence of match results.
The plots show the Arena score difference between the two players after X games. It shows how quickly the original Elo difference is "forgotton". At the beginning, the difference is 100. With each equal match result, both Arena scores converge closer together, the difference gets smaller.
K=40 (current Arena system)
K=20 (current Elo system)
K=10
K=5
In general, this is expected/wanted behavior. If two players get the exact same results against the exact same opponents, then their Elo/Arena score should converge to the exact same value. The problem is how quickly this occurs. With a high K factor, the convergence is super quick and the original Arena score difference of 100 is nullified quickly. As pointed out, this is stupid for games with a strong luck component.
With K=40, the original Arena score difference of 100 is reduced by half already after only about 15 games. After 30 games, the difference is only at 20, meaning that 80% of historic performance difference is already forgotten after 30 games. (Note that it doesn't matter much what initial Elo difference I start the simulation with. Had I set a difference of 200 in the beginning, still about 50% of this difference is gone after 15 games, about 80% of the difference is gone after 30 games.)
There you have it, with the current Arena system, your score is mostly dependent on your last ~30 games, with stronger weight for the more recent games.
With K=20, it looks already much more reasonable, but the convergence is still way too quick for luck-heavy games. If there is a lot of luck, you need a longer series of games actually matter for the Elo/Arena score to get a somewhat meaningful performance score.
K=10 is something I could live with for games like 7Wonders, Railroad Ink, Harmonies. For games like Can't Stop or Lucky Numbers, you could even choose K=5. The effect is that less recent games are not downgraded as quickly for the Elo/Arena score, less sensitivity on the latest result.
Of note: This is for 2p games. Higher player counts are more complicated.
If you are interested in (not the best) R code that does the simulations, feel free to reach out to me.
the already high usual Elo K factor for the Arena mode. It makes the Arena score fluctuate so severely that it loses any meaning.
