Craft loses contact with database, and stops daemon

Question

Somehow the queue/listen occasionally loses access to the database:

FATAL     Exited too quickly

.

Database Connection Error: Craft CMS can’t connect to the database.

As the connection is lost, it seems like the daemon is not able to recover, and needs to be manually restarted.

Additional info:

• Database: MySQL 8.0.35
• There was no deployment being done at the time of failure.
• I don’t currently have any automatic restarting of the daemon after deployments.
• There are no other indications of the database being down at the time the daemon dropped out.
• It has happened across multiple sites, and across hosts.
• One common factor is that Laravel Forge is used for provisioning and setting up the Daemon.
• It has happened to custom VPS’s, and Digital Ocean Droplets.
• All affected sites as far as I know are Craft 4.5.5 or later.
• This issue is new to me the last 3-5 months, even though I've made Craft sites with this same setup for a couple of years.
• The issue has also been discussed in the Discord channel "Devops" late August 2023. https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099
• It seems to have a tendency to happen at night time. Which is also roundabout the same time doing database backups. It doesn't correlate greatly, so it seems farfetched to be linked. Often linked by +- 2 hours.

Most of the time this happens, its quite critical to get up and running again, and I don't know how to really get to the bottom of the problem.

Daemon Supervisor Configuration

[program:daemon-758141]
command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/www.site.com/craft queue/listen --verbose

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=2
startsecs=1
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-758141.log
stopwaitsecs=10
stopsignal=SIGTERM
stopasgroup=true
killasgroup=true

Answer 1

What about the unattended-upgrades?

We had a similar problem with daemons (and still do) but not with the database. In our case, whenever an update occurs, with some chance daemons die with the same error(FATAL Exited too quickly). The supervisor is restarted, but somehow the daemon doesn’t seem to work.

To test this theory, look at when specific service (supervisor, mysql etc) crashed and compare it with the unattended-upgrade.log(everything should be in the /var/log/)

E.g. https://www.reddit.com/r/docker/comments/k4kk97/docker_daemon_crash_with_unattendedupgrades/

Answer 2

We're also seeing this issue on Laravel Forge. For us it mostly happens after a Craft update is deployed, so it might have something to do with the queue runner still having the previous Craft version loaded.

While I'm not sure what causes it, as a workaround you can ensure that the queue listener restarts periodically and/or after deployments.

There are some problems with this approach. For sites with a lot of traffic and queue jobs, restarting the queue runner may result in the queue getting backed up. Also, when a queue job is running while the queue runner is being forcefully restarted, it can cause the job to fail. To address this, we also have a nightly cron job that runs queue/retry to automatically retry failed queue jobs. But for small and medium sites that don't have tons of queue jobs coming in 24/7, this approach works quite well.

Restarting the queue runner periodically

Create a cron job in Forge, in the server settings under Scheduler:

• Command: sudo supervisorctl restart all
• User: forge
• Frequency: Nightly (or choose whatever you like)

On newly provisioned servers, this should work without any further adjustments, because Forge now allows the forge user to execute supervisorctl commands without a password. On older servers, make sure the following snippet is in your /etc/sudoers.d/supervisorctl:

forge ALL=NOPASSWD: /usr/bin/supervisorctl reload
forge ALL=NOPASSWD: /usr/bin/supervisorctl reread
forge ALL=NOPASSWD: /usr/bin/supervisorctl restart *
forge ALL=NOPASSWD: /usr/bin/supervisorctl start *
forge ALL=NOPASSWD: /usr/bin/supervisorctl status *
forge ALL=NOPASSWD: /usr/bin/supervisorctl status
forge ALL=NOPASSWD: /usr/bin/supervisorctl stop *
forge ALL=NOPASSWD: /usr/bin/supervisorctl update *
forge ALL=NOPASSWD: /usr/bin/supervisorctl update

Restarting the queue runner after deployments

To restart the queue runner after every deployment, add this to the end of your deployment script:

sudo supervisorctl restart all

This will restart all daemons. Alternatively, you can also find the ID of your queue runner (use sudo supervisorctl status to list all active daemons) and only restart that in your deployment (replace <daemon-id> with the actual ID):

sudo supervisorctl restart <daemon-id>

On newer servers, if you're not using website isolation, this should work by default. If you are using website isolation, you need to allow the website user to run supervisorctl without a password, since you can't (or rather, shouldn't) provide the password in the deployment script.

Add this to your /etc/sudoers.d/supervisorctl:

websiteuser ALL=NOPASSWD: /usr/bin/supervisorctl *

Replace websiteuser with the website's (Ubuntu) user in Forge.