Yesterday, we performed an important server capacity upgrade on Board Game Arena.
The service was okay up to ~1100 online playing players, but above this number it was getting slower and slower, with too many of "BGA mainsite does not respond" or "Server update in progress" or "Error with notification system".
For those who are interested by a little "behind the scene", here is the explanation about why it took so long before we react, what we've done and why it was difficult.
More servers = more capacity?
BGA is quite different from many other websites, because it's mostly based on "realtime" : you don't have to refresh your browser to see that someone created a table, or that your opponent has played.
The architecture needed to do this is way more complex than for a standard website. Basically, for a standard website, when you want to upgrade your service capacity, you just need to add more servers. More servers = more capacity. Seems pretty straightforward.
The realtime server
This is also partially true for BGA, and regularly we are adding more servers. But this is not the case for the "realtime" server, the one which allows you to receive changes in realtime from BGA. This is this famous "notification service" that sometimes gives you horrible red error messages.
So why can't we add more "realtime servers"?
Because everyone of you has to be connected to the same realtime server. Otherwise, when you are doing a move, your opponent won't be notified about it. So we are using a specific technology (NodeJs) that is allowing many many players to be connected at the same time to the same realtime server. And this is the situation from the beginning of BGA.
Weeks ago, as BGA was getting more are more audience, and the realtime server wasn't able to handle this at peak hours. This server was already the most powerful available server, so it become obvious that we reached the limit of the "classic" approach.
Grow and multiply
So here is what we've done : we splitted the realtime server in 4 realtime servers - which (theoretically) multiply the server realtime capacity by four.
To do this, we had to make sure that these 4 servers are synchronized, so if you and your opponent are connected to different servers, you can still play together. We had to rewrite 40-50% of the realtime server logic, which is not a "big" work but this is very sensitive one. Basically if something goes wrong here, it can break the whole service in seconds.
The day after
Today, the service seems stable, and most of the issues we found yesterday has been solved. However, we changed a lot of things in a small amount of time, so there may be some remaining issues.
If you found one of these issue, please report it in the bug system or as a comment on this post.
Note : if your issue has been reported months ago, this is probably not an issue related to the yesterday's operations.
Thank you everyone for making this service popular, and sorry about the impact of these - needed - changes on the service.
... and by the way, to immediately use our increased realtime capacity, we added a "X is writing something ..." feature on the chat
The service was okay up to ~1100 online playing players, but above this number it was getting slower and slower, with too many of "BGA mainsite does not respond" or "Server update in progress" or "Error with notification system".
For those who are interested by a little "behind the scene", here is the explanation about why it took so long before we react, what we've done and why it was difficult.
More servers = more capacity?
BGA is quite different from many other websites, because it's mostly based on "realtime" : you don't have to refresh your browser to see that someone created a table, or that your opponent has played.
The architecture needed to do this is way more complex than for a standard website. Basically, for a standard website, when you want to upgrade your service capacity, you just need to add more servers. More servers = more capacity. Seems pretty straightforward.
The realtime server
This is also partially true for BGA, and regularly we are adding more servers. But this is not the case for the "realtime" server, the one which allows you to receive changes in realtime from BGA. This is this famous "notification service" that sometimes gives you horrible red error messages.
So why can't we add more "realtime servers"?
Because everyone of you has to be connected to the same realtime server. Otherwise, when you are doing a move, your opponent won't be notified about it. So we are using a specific technology (NodeJs) that is allowing many many players to be connected at the same time to the same realtime server. And this is the situation from the beginning of BGA.
Weeks ago, as BGA was getting more are more audience, and the realtime server wasn't able to handle this at peak hours. This server was already the most powerful available server, so it become obvious that we reached the limit of the "classic" approach.
Grow and multiply
So here is what we've done : we splitted the realtime server in 4 realtime servers - which (theoretically) multiply the server realtime capacity by four.
To do this, we had to make sure that these 4 servers are synchronized, so if you and your opponent are connected to different servers, you can still play together. We had to rewrite 40-50% of the realtime server logic, which is not a "big" work but this is very sensitive one. Basically if something goes wrong here, it can break the whole service in seconds.
The day after
Today, the service seems stable, and most of the issues we found yesterday has been solved. However, we changed a lot of things in a small amount of time, so there may be some remaining issues.
If you found one of these issue, please report it in the bug system or as a comment on this post.
Note : if your issue has been reported months ago, this is probably not an issue related to the yesterday's operations.
Thank you everyone for making this service popular, and sorry about the impact of these - needed - changes on the service.
... and by the way, to immediately use our increased realtime capacity, we added a "X is writing something ..." feature on the chat
