Page 1 of 1
Studio server down
Posted: 03 January 2021, 04:27
by paramesis
I've been getting a 504 Gateway Time-out error on all open tables of all games for the last hour.
I have been able to create and express stop tables, but not open any of them. Is this due to planned maintenance?
If not, there's a possible reason that would be concerning. The problem began immediately after I took an action which I now realize may have caused an infinite loop. On the previous studio server, this would produce a fatal error and halt after 1 second of execution. Is it possible that the execution timeout wasn't configured with the new server, and my infinite loop is still going, causing the studio server not to respond for anybody?
I've called Express Stop on that table, but all other tables are still experiencing the error.
Re: Studio server down
Posted: 03 January 2021, 06:44
by paramesis
I was able to use the server again after a few hours. Given the exact timing of the issue, the infinite loop seems like a possible cause. Is there an execution timeout on the Studio server?
Re: Studio server down
Posted: 03 January 2021, 09:21
by Geoffster
I was penalised due to "504 error". Is there a way to request the penalty overturned or cancelled?
Re: Studio server down
Posted: 03 January 2021, 17:04
by Een
paramesis wrote: ↑03 January 2021, 06:44
I was able to use the server again after a few hours. Given the exact timing of the issue, the infinite loop seems like a possible cause. Is there an execution timeout on the Studio server?
The studio web server has not changed recently (only the database server).
So there is a php timeout.
I'll check the logs.
Geoffster wrote: ↑03 January 2021, 09:21
I was penalised due to "504 error". Is there a way to request the penalty overturned or cancelled?
On the studio, account reputation is not important, those are test accounts.
Re: Studio server down
Posted: 03 January 2021, 18:06
by Een
I see database timeouts for your table 2322990. Not sure what happened exactly, but it looks more like a database lock on this table than a server wide issue.
Re: Studio server down
Posted: 04 January 2021, 23:14
by paramesis
Een wrote: ↑03 January 2021, 18:06
I see database timeouts for your table 2322990. Not sure what happened exactly, but it looks more like a database lock on this table than a server wide issue.
Some other developers have experienced similar symptoms after an infinite loop. Is it possible a database lock is preventing the php timeout from functioning normally? I never saw the timeout error, just a connection error "The server is not responding, are you connected?..." and was then unable to open any studio table for a few hours.
Re: Studio server down
Posted: 05 January 2021, 09:02
by lordalx
Hi.
Several and repeatable occurs on m'y side as well.
It don't lock only the current table but prevent any connexion to the studio.
Re: Studio server down
Posted: 05 January 2021, 09:21
by Een
I found just one php timeout in the logs (for player actions, it's actually longer than just 1 second)
Indeed, external calls (such as database) are not counted by php for the timeout since it's not actually php running time.
There were not so many database timeouts either (maybe 10 or something like that).
Usually, infinite loops hit the memory allocation limit rather than the execution limit, but of course, it all depends on what was in the loop. Potentially if the loop as very little php running time, very long database query time or ext calls time, and doesn't allocate memory, I suppose it can run for a long time.
Still, it only blocks a thread of the web server, so other developers are fine (and I think other games timeout for you because we probably still have a config limiting to one current request per IP on the studio; we can look into lifting this as we lifted it for the mainsite)