About November 25th service interruption and issues

[sourisdudesert]

On Friday, November 25th, starting 8:30 CET, we did a major maintenance operation on BGA.

It caused the following :

• Service interruption between 8:30 CET and 12:30 CET.
• Bugs on leaderboards (wrong rankings, wrong "most victories", wrong "most games played", ...).
• Bugs on card games : cards shuffling was always producing the same results (particularly visible on Love Letter, Belote and Hearts).
• Game replay was broken during few hours.

The reason of this maintenance operation was to do a general update of operating systems and softwares used by BGA servers, in order to increase this system security, reliability and performances (see details below for "technicians"). The goal was also to fix the "players marked as online while they are offline" bug that is visible on the platform since few weeks.

The planned duration for this operation was 2h30. Unfortunately, the realtime component of BGA failed to work after the upgrade. More precisely it was a little unstable, with a strange behavior. This was unexpected because we tested this upgrade on our computers without problem. Then we decided to reinstall it from scratch to be 100% sure there would be no issue. This took us an additional 2h at the end of the process, and explains the (very long) service interruption.

We're of course sorry for this, especially for the tournament that was planned during the service interruption.

The bugs on leaderboard and card games has been fixed around 18:00 CET. Game replays has been fixed around 21:00 CET. The situation is now stable.

I know that these operations are painful for everyone, and that you can only see the downside of it - it always break something - but this is really needed to keep BGA software up to date, safe and reliable.


Details for technicians and curious :

• We've upgraded all our servers to the latest Linux distribution & kernel
• We've upgraded our databases to latest MySql version.
• We've upgraded PHP to the new "PHP 7" version.
• The 2 majors bugs (leaderboards and cards shuffling) was caused by an unexpected behavior of user-defined DB variables from the latest version of MySql, that was not easy to fix because we were forced to rewrite completely the code corresponding to cards shuffling/leaderboards rankings, while BGA was already running. We are not 100% satisfied by the result so we have to work more on this, but at least this is fixed now.

Thanks for your support & understanding.

[yoyote]

like always, thanks for your communication.
The fact is, now I know I'm an addict

[Pedros]

> yoyote - yes, it was a very long morning!

And thanks again guys

[Willie333b]

Thanks, but pretty sad yesterday.

[Murlonioteam]

Thx for the explanation!
We all glad the Minotaurus is lock again,,,,

[travbot]

Thanks for all the hard work to make this site a great place to game!

[wyofan87]

Thank you for the fixes.

[ChewOnIt]

It is OK, thanks for the hard work, everyone

[raphpremier]

Yes thank you for your work, for the explanation.

BGA is so great!

[JoshTriplett]

Bugs on card games : cards shuffling was always producing the same results (particularly visible on Love Letter, Belote and Hearts).

I'm curious how the maintenance caused that to happen. How does the shuffling process work that would allow this result?