Anyone else getting "504 Gateway Time-out" errors?

[JonChambers]

For what it's worth, I often find the studio is slow and spottily-responsive when working on my projects late at night - around mindight/1am-ish, UK time. It's generally OK before then (between say 8 and 12 - I don't often do much in the studio at other times of day or night so can't comment on that).

But by this I just mean that when making a move it takes forever to respond, and sometimes gives be the "are you connected to the server" red banner - ironically quite often the move registers and the interface updates just after that (but not always, sometimes I have to resend the move, not infrequently several times in a row). That's different from seeing a 504 error, which I've never noticed myself. So probably not the same thing at all.

Having said that, I'm concerned about @Een's responses, that a dev could basically bring down the whole Studio with malicious code (whether deliberately or accidentally malicious). Granted hackers probably won't bother with BGA studio, but then it seems that no "hacking" is actually involved - you would just have to register an account, start a project and stick in a "while (true) { }" in setUpNewGame, start a game and walk away. Granted it's probably easy for admins to spot and shut down obvious stuff like that but it's a bit worrying that it's so easy to kill the server.

I'd just like to be able to see a graph of how busy the studio is/was. It'd be nice to see the daily trend instead of just relying on experience.

For example, ten minutes ago, Een says the server load is low.

Right now... my experience is that it is anything but. Did something change in ten minutes? Or is this a bug on my end that only appears to be server related? Or have I unknowingly caused the high server load with my extremely demanding game? I have no data to check on to answer these questions.

Though I'll be cautious about being "concerned" about a website lacking ideal features. Sure, it would be nice to have a system where the number of devs logged in each get 1/n of the server hardware, but I don't want to be blind about the "nice to haves" that I already have without having to pay for any of it.

I've complained plenty about the website having features I wish I could turn off, but lacking features I want to turn on... that's a bigger ask.

[Een]

For what it's worth, I often find the studio is slow and spottily-responsive when working on my projects late at night - around mindight/1am-ish, UK time. It's generally OK before then (between say 8 and 12 - I don't often do much in the studio at other times of day or night so can't comment on that).

Thanks for the feedback about specific slow times of the day.

Having said that, I'm concerned about @Een's responses, that a dev could basically bring down the whole Studio with malicious code (whether deliberately or accidentally malicious). Granted hackers probably won't bother with BGA studio, but then it seems that no "hacking" is actually involved - you would just have to register an account, start a project and stick in a "while (true) { }" in setUpNewGame, start a game and walk away. Granted it's probably easy for admins to spot and shut down obvious stuff like that but it's a bit worrying that it's so easy to kill the server.

With the studio, we do allow devs to write code and run it (as you know as a studio developer), so yes, that comes with a tradeoff. We think that we have to accept it to enable developer contributions, and that it's acceptable since the studio is separate from prod. What you describe anyway is functionally equivalent to a DDoS that is always a possibility for any website, and potentially easier to solve for admins.

[Een]

I'd just like to be able to see a graph of how busy the studio is/was. It'd be nice to see the daily trend instead of just relying on experience.

We don't have that at the moment, it's on my list but I have a big list and it's not near the top.

On a side note, looking at the system side of things, working threads on apache are from your game, so the issue may come from your game. Can't investigate more right now, sorry.

[robinzig]

With the studio, we do allow devs to write code and run it (as you know as a studio developer), so yes, that comes with a tradeoff. We think that we have to accept it to enable developer contributions, and that it's acceptable since the studio is separate from prod.

Of course! I think I was being dense, assuming that each game on studio ran on its own little "sandbox" and it was (or should be) impossible to write code that affected the rest of the server, ie that if you cause an infinite loop in your game then only your game would be affected - but of course as it's all running on the one server you're right that there's nothing to stop a malicious developer essentially DDOSing the server if they felt so inclined. Not sure why I didn't realise this.

Thanks for the reply

[JonChambers]

On a side note, looking at the system side of things, working threads on apache are from your game, so the issue may come from your game. Can't investigate more right now, sorry.

That would be a big help. I don't know what the back end of these systems look like, but if you somehow have access to the line number where the loop is repeating from, that will save me hours of work investigating.

Normally, errors show up straight away, so it's easy to spot. This one seems to linger, as if I've got lots of processes somehow running in the background, but I have no way of seeing what's going on in the background.

I AM testing the 6 player version, and I have several other tables open, so maybe it's just normal load, but I can't imagine a website of this size would struggle to handle 6 players at once, so I've got no idea. The more information you can give me, the better.

[JonChambers]

Another minor update:

I did a text search for "for(" and found a loop where I tested using "!=" instead of "<". So if one of the values was not a number, that would have produced an infinite loop. (I think. I've never tried it intentionally.) Maybe that was it?

I don't know, because I'm still 504ing. Maybe I've caused countless threads running, crashing the server for everyone else? If so... sorry.

[JonChambers]

Another update:

Updating the script and it working now is probably a coincidence, but everything is working great now! (If it was the script, it should have started working yesterday when I changed it, and it would still be throwing an error now, which it isn't.)

So frustrating yesterday because it was going to be the FIRST TIME I ever played the 6p version to completion and CRASH!!! So, I finished the game from yesterday, it runs perfect! So damn proud!

I know this is off topic, but I'm so damn proud I'm gonna type it anyway. I'm nearly finished. I'm nearly ready for launch. Here's my to-do list:

• Fix a minor persistence bug that shows irrelevant data from previous round that should be blank.

• Write some statistics for 3-6 players (2 players complete)

• Write progression for 3-6 players (2 players complete)

• Create a banner image and action shots

• Zombie moves

• Spectator mode (this will be harder than everything else combined)

Then I'm finally done! Tears of joy.

Back on topic... yeah, whatever that thing was, it's working now. I'll probably never know.

[Een]

I did a text search for "for(" and found a loop where I tested using "!=" instead of "<". So if one of the values was not a number, that would have produced an infinite loop. (I think. I've never tried it intentionally.) Maybe that was it?

It looks like a good candidate explanation.

Maybe I've caused countless threads running, crashing the server for everyone else? If so... sorry.

Could be, but from this thread no other devs have reported 504s, so it seems to have impacted mostly yourself (which is logical if the issue was in your code). Anyway, it happens.

[robinzig]

• Spectator mode (this will be harder than everything else combined)

I don't want to open a can of worms - but why do you expect this to be so hard? All you should have to do is hide any areas of the screen that show information that is private to one or more players. I don't really see how it will ever be more than a few lines of CSS, and perhaps a tiny bit of JS too in some cases. While I know nothing about your game, I do not see how any game would need to do anything drastic here.

Even if you do want to do something complex and custom, I suspect it's wasted effort as I don't think most BGA games attract many spectators!

[JonChambers]

• Spectator mode (this will be harder than everything else combined)

I don't want to open a can of worms - but why do you expect this to be so hard? All you should have to do is hide any areas of the screen that show information that is private to one or more players. I don't really see how it will ever be more than a few lines of CSS, and perhaps a tiny bit of JS too in some cases. While I know nothing about your game, I do not see how any game would need to do anything drastic here.

Even if you do want to do something complex and custom, I suspect it's wasted effort as I don't think most BGA games attract many spectators!

I was expecting this exact response, but you phrased it with more humility than I expected.

The entire layout of the game responds to the role you play for that turn. There is a LOT of "so, what's your role?" in the code, and I need to come up with an answer for "spectator" in every single situation.

There's even a script I wrote that displays a message custom to the gamestate and your role in the game. But they're all worded in a way that assumes you are playing.

It's a mad libs of different sentences smashed together, but the most complex of them has 14 different strings individually written. So now I need to write even more messages to the effect of, "Now that was a good move! When you finally get around to playing this game, you should make moves like that!" (It won't be exactly that message, but I'm trying to keep my answer generic to all games rather than specific to mine.)

It's not just spectator mode being hard. It's everything else on the list being easy!