MAJOR OUTAGE caused by daylight saving bug

[quietmint]

Both production BGA and studio are totally broken right now because of the following error. Looks like this SQL runs when I try to login or start a table in studio (maybe also at every move? but I can't get that far to test).

Google says Daylight saving time 2021 in Europe will begin at 1:00 AM on Sunday, March 28 so the error happens because now + 3 months results in a time that doesn't exist. Using DATE_ADD() apparently is not safe, unless you set the MySQL database's timezone to UTC (so there's no daylight saving involved).

Code: Select all

Unexpected error:  Error while processing SQL request (Localhost via UNIX socket): INSERT INTO l_catchup_email_metrics (lcm_user_id, lcm_next_send_time) VALUES (2305326, DATE_ADD(NOW(), INTERVAL 3 MONTH)) ON DUPLICATE KEY UPDATE lcm_next_send_time = VALUES(lcm_next_send_time)
Incorrect datetime value: '2021-03-28 02:01:27' for column 'lcm_next_send_time' at row 1

[Absurd57]

Actually that means that bug will be automatically fixed in 9 minutes (in 2 am)

[quietmint]

Yes, the hour has changed and the issue is no longer happening, but it must still be fixed. This caused an entirely-preventable outage that took the entire site offline for a prime hour during Christmas weekend (in the US).

Besides causing a problem next December, there may be a problem in July when daylight saving time ends (when a time like 2021-10-31 02:01:27 is ambiguous because the 2 o'clock hour repeats).

[sourisdudesert]

Hi.

Sorry about that. I confirm the cause: this will be fixed shortly.