Service disruptions starting from November 28th

[lordalx]

As many of you noticed, we faced some significant disruptions starting last Friday.

The issue originated with one of our providers, leading to a severe snowball effect since Friday evening (CET). We’ll provide more details on Monday, but the service experienced several outages and was particularly unstable for real-time games. Our tech team worked hard throughout the weekend, even late into the night, to mitigate the situation.

Rest assured, penalties will be removed automatically, so there’s no need to contact support about this.

Thanks for you patience

[Laurens de Graaf]

Thanks for the update!

[Sequim Grandma]

I switched from my tablet to my phone to see if the service was better and my IP got banned. Will that get fixed too?

[ab in dc]

I was in the middle of a real-time game when the board crashed this morning. I was able to switch it to turn-based, but my profile still says it's real-time and now i'm still locked out of starting new real-time games. Any way to fix that?
Table 768055860

[xate]

I switched from my tablet to my phone to see if the service was better and my IP got banned. Will that get fixed too?

I have unbanned your IP. Sorry for the inconvenience.

[BalGAmor]

It’s been quite the weekend, right...?

Starting on Friday afternoon (EU time), our main hosting provider suffered from significant network instability. Our real-time system, which powers connections for games, chat, and player presence, was heavily affected. Because these connections must remain stable to function properly, even small interruptions can trigger a chain reaction of disconnects and noticeable latency across the site.

To prevent the platform from staying unstable for too long, we attempted several emergency solutions throughout the weekend. We first migrated services to servers from another provider, and while this worked for a while, the setup wasn’t robust enough to handle the full live workload.

We then moved back to our regular provider, but the same network issues to resurface a few hours later and we went down again on Friday evening.

Next, we tried another datacenter, which appeared stable… until it began showing the exact same symptoms on Sunday afternoon.

When one thing was fixed, another broke down making it a long and painful process that lasted the whole weekend.

Long because we never expect a reactive maintenance to last more than a couple of hours. Painful because the fix wasn’t an easy one and, as players ourselves, we don’t like to see the site down not being able to play our favorite games.

After multiple migrations to various services, things finally resumed to normal on Sunday evening.

Rest assured that our team is always doing their best to get things working properly, that we are monitoring the evolution of the situation and looking into the best options to prevent such an event from happening again.

We would like to thank you for your patience and understanding.

[katmk]

Thank you for providing this explanation, I definitely appreciate knowing what the situation has been.

[covilha]

Rest assured, penalties will be removed automatically, so there’s no need to contact support about this.

Any idea when this will happen?

So we know when to verify if the penalties were correctly removed.

Thank you!

[Meeplelowda]

Rest assured that our team is always doing their best to get things working properly, that we are monitoring the evolution of the situation and looking into the best options to prevent such an event from happening again.

We would like to thank you for your patience and understanding.

To clarify, when BGA says "penalties" will be removed, do they mean that: 1) karma lost from a penalty will be restored, 2) additionally, the record on your profile of the karma penalty will also be removed, and 3) that the game in which the penalty was incurred will be neutralized as well, meaning the impact on Elo will be neutralized? I know it's at least 1. But is it: 1; 1 and 2; 1 and 3; or 1, 2, and 3?

I ask because the post right above mine is from someone with 100% karma, so we know either 1 has already been applied or they otherwise restored their karma, so they must be complaining about something else besides 1.

[Jellby]

Unfortunately, I think "penalties" refer to karma penalties, not to the normal elo you lose due to losing a game, and (I believe) not even to the extra -10 elo for quitting a game in progress.

You can see it says here that your (karma) penalty is already cancelled:
https://boardgamearena.com/table?table=767294811

PS. I'm pretty sure it's just 1. The profile still has "1 quit", and the elo score matches all the game changes. Applying 3 would be impossible anyway. It would mean (a) removing elo from players who "won" the game (imagine the outcry), and (b) retroactively modifying the elo changes of all the subsequent games which should have been done with the corrected elo...