About turn-based May 3rd major issue
Posted: 04 May 2016, 08:16
Hello,
On May 3rd, those of you who are playing BGA in turn-based mode encountered a major problem with some games marked as "abandoned automatically because no one is playing for a long time".
This is by far the biggest BGA accident for the past 5 years, and we're sorry about this.
What happened
From one month (announced here), we were preparing to evolve BGA into a secured web site (https).
Any change moving forward has got some risk bundled with it and this was no exception. Today after the testing period, things were looking good, so we activated automatic redirect to https... and then... we got a server crash. And with it through an hazardous chain of events that still amazes us, exchanges between the main site and the game servers caused a lot of the turn-based games "last move" timestamps to be moved into the past, and being considered obsolete... the games were closed automatically starting 16:30 CET yesterday, and in only few minutes 6000 turn-based games has been closed.
Basically, it was a combination of a technical failure, an old and risky piece of code that we shouldn't have written, and bad luck.
What we did
As soon as we were alerted, we made sure that no additional tables can be affected by this issue.
I was in charge of fixing the issue, while Een first mission was to communicate with you and answer your questions.
Then, we worked on finding what caused the issue. Around 19:30 CET we found out the unlikely chain of events that caused the issue. We worked immediately on a fix, and at 20:30 CET we publish an update of the website with a fix and some additional security measure to make sure this kind of accident can't happen again in the future.
After that, we start to study how we could "bring back to life" all the abandoned game. This is something that we were unprepared to do : a game finished is a game finished, and there is no way it can be reverted to a playing game. After analysis, we found out that for 95% of tables there were no safe way to go back to the state corresponding to time just before the incident.
However, we found another solution that could bring back all games only a few moves to the past, based on the "game replay" feature which is absolutely not designed for such a usage. We start working on this solution around 21:30 CET and a first series of tables were back in your "games in progress" tab around 22:30 CET.
The whole process finished at 03:00 CET today.
Note : We removed all penalties received during the whole period.
What are the consequences for the games
Depending on the games, between 0 and 9 of the last moves has been cancelled (like a "jump in the past"), with an average of 4 moves.
Note : A "move" is not a turn : if you are doing several action during a turn, there are several moves.
Some of games couldn't be restored because only very few moves were played - but our first priority was to save games that were running for a long time.
About turn-based tournament : unfortunately, we couldn't remove additional game results generated by the abandoned games. Depending on the tournament system, this could break completely the tournament or just add some "0.5" points to every participant.
Can I play now?
Sure! The situation is now stable. Please advise us immediately (contact (at) boardgamearena.com) if you see anything strange - except from the little "jump in the past".
May the 4th be with you.
On May 3rd, those of you who are playing BGA in turn-based mode encountered a major problem with some games marked as "abandoned automatically because no one is playing for a long time".
This is by far the biggest BGA accident for the past 5 years, and we're sorry about this.
What happened
From one month (announced here), we were preparing to evolve BGA into a secured web site (https).
Any change moving forward has got some risk bundled with it and this was no exception. Today after the testing period, things were looking good, so we activated automatic redirect to https... and then... we got a server crash. And with it through an hazardous chain of events that still amazes us, exchanges between the main site and the game servers caused a lot of the turn-based games "last move" timestamps to be moved into the past, and being considered obsolete... the games were closed automatically starting 16:30 CET yesterday, and in only few minutes 6000 turn-based games has been closed.
Basically, it was a combination of a technical failure, an old and risky piece of code that we shouldn't have written, and bad luck.
What we did
As soon as we were alerted, we made sure that no additional tables can be affected by this issue.
I was in charge of fixing the issue, while Een first mission was to communicate with you and answer your questions.
Then, we worked on finding what caused the issue. Around 19:30 CET we found out the unlikely chain of events that caused the issue. We worked immediately on a fix, and at 20:30 CET we publish an update of the website with a fix and some additional security measure to make sure this kind of accident can't happen again in the future.
After that, we start to study how we could "bring back to life" all the abandoned game. This is something that we were unprepared to do : a game finished is a game finished, and there is no way it can be reverted to a playing game. After analysis, we found out that for 95% of tables there were no safe way to go back to the state corresponding to time just before the incident.
However, we found another solution that could bring back all games only a few moves to the past, based on the "game replay" feature which is absolutely not designed for such a usage. We start working on this solution around 21:30 CET and a first series of tables were back in your "games in progress" tab around 22:30 CET.
The whole process finished at 03:00 CET today.
Note : We removed all penalties received during the whole period.
What are the consequences for the games
Depending on the games, between 0 and 9 of the last moves has been cancelled (like a "jump in the past"), with an average of 4 moves.
Note : A "move" is not a turn : if you are doing several action during a turn, there are several moves.
Some of games couldn't be restored because only very few moves were played - but our first priority was to save games that were running for a long time.
About turn-based tournament : unfortunately, we couldn't remove additional game results generated by the abandoned games. Depending on the tournament system, this could break completely the tournament or just add some "0.5" points to every participant.
Can I play now?
Sure! The situation is now stable. Please advise us immediately (contact (at) boardgamearena.com) if you see anything strange - except from the little "jump in the past".
May the 4th be with you.