Page 1 of 2

About November 25th service interruption and issues

Posted: 26 November 2016, 12:48
by sourisdudesert
On Friday, November 25th, starting 8:30 CET, we did a major maintenance operation on BGA.

It caused the following :
  • Service interruption between 8:30 CET and 12:30 CET.
  • Bugs on leaderboards (wrong rankings, wrong "most victories", wrong "most games played", ...).
  • Bugs on card games : cards shuffling was always producing the same results (particularly visible on Love Letter, Belote and Hearts).
  • Game replay was broken during few hours.
The reason of this maintenance operation was to do a general update of operating systems and softwares used by BGA servers, in order to increase this system security, reliability and performances (see details below for "technicians"). The goal was also to fix the "players marked as online while they are offline" bug that is visible on the platform since few weeks.

The planned duration for this operation was 2h30. Unfortunately, the realtime component of BGA failed to work after the upgrade. More precisely it was a little unstable, with a strange behavior. This was unexpected because we tested this upgrade on our computers without problem. Then we decided to reinstall it from scratch to be 100% sure there would be no issue. This took us an additional 2h at the end of the process, and explains the (very long) service interruption.

We're of course sorry for this, especially for the tournament that was planned during the service interruption.

The bugs on leaderboard and card games has been fixed around 18:00 CET. Game replays has been fixed around 21:00 CET. The situation is now stable.

I know that these operations are painful for everyone, and that you can only see the downside of it - it always break something - but this is really needed to keep BGA software up to date, safe and reliable.


Details for technicians and curious :
  • We've upgraded all our servers to the latest Linux distribution & kernel
  • We've upgraded our databases to latest MySql version.
  • We've upgraded PHP to the new "PHP 7" version.
  • The 2 majors bugs (leaderboards and cards shuffling) was caused by an unexpected behavior of user-defined DB variables from the latest version of MySql, that was not easy to fix because we were forced to rewrite completely the code corresponding to cards shuffling/leaderboards rankings, while BGA was already running. We are not 100% satisfied by the result so we have to work more on this, but at least this is fixed now.
Thanks for your support & understanding.

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 13:22
by yoyote
like always, thanks for your communication.
The fact is, now I know I'm an addict :)

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 13:56
by Pedros
> yoyote - yes, it was a very long morning!

And thanks again guys

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 14:02
by Willie333b
Thanks, but pretty sad yesterday.

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 14:13
by Murlonioteam
Thx for the explanation!
We all glad the Minotaurus is lock again,,,,

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 14:47
by travbot
Thanks for all the hard work to make this site a great place to game!

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 14:52
by wyofan87
Thank you for the fixes.

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 16:10
by ChewOnIt
It is OK, thanks for the hard work, everyone :D

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 17:05
by raphpremier
Yes thank you for your work, for the explanation.

BGA is so great!

Re: About November 25th service interruption and issues

Posted: 26 November 2016, 23:08
by JoshTriplett
Bugs on card games : cards shuffling was always producing the same results (particularly visible on Love Letter, Belote and Hearts).
I'm curious how the maintenance caused that to happen. How does the shuffling process work that would allow this result?