6

Somehow the queue/listen occasionally loses access to the database:

FATAL     Exited too quickly

.

Database Connection Error: Craft CMS can’t connect to the database.

As the connection is lost, it seems like the daemon is not able to recover, and needs to be manually restarted.

Additional info:

  • Database: MySQL 8.0.35
  • There was no deployment being done at the time of failure.
  • I don’t currently have any automatic restarting of the daemon after deployments.
  • There are no other indications of the database being down at the time the daemon dropped out.
  • It has happened across multiple sites, and across hosts.
  • One common factor is that Laravel Forge is used for provisioning and setting up the Daemon.
  • It has happened to custom VPS’s, and Digital Ocean Droplets.
  • All affected sites as far as I know are Craft 4.5.5 or later.
  • This issue is new to me the last 3-5 months, even though I've made Craft sites with this same setup for a couple of years.
  • The issue has also been discussed in the Discord channel "Devops" late August 2023. https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099
  • It seems to have a tendency to happen at night time. Which is also roundabout the same time doing database backups. It doesn't correlate greatly, so it seems farfetched to be linked. Often linked by +- 2 hours.

Most of the time this happens, its quite critical to get up and running again, and I don't know how to really get to the bottom of the problem.

Daemon Supervisor Configuration

[program:daemon-758141]
command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/www.site.com/craft queue/listen --verbose

process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=forge
numprocs=2
startsecs=1
redirect_stderr=true
stdout_logfile=/home/forge/.forge/daemon-758141.log
stopwaitsecs=10
stopsignal=SIGTERM
stopasgroup=true
killasgroup=true
3
  • 1
    This seems like a rather complicated bug based on the lengthy Discord discussion, but I'd like to just leave these links here as they are in my documentation for an issue that I fixed which might possibly be related. discord.com/channels/456442477667418113/456442477667418115/… freedesktop.org/software/systemd/man/… unix.stackexchange.com/questions/289629/… Commented Nov 3, 2023 at 11:24
  • 1
    Basically, with the default settings, even a 500ms DB outage would have caused the queue daemon to fail and not restart. Commented Nov 3, 2023 at 11:25
  • Thanks for good links and points. Unfortunately they seem to be only systemd related, but still gives some guidance. In my case Supervisor is used, and not systemd, and I dont seem to find a similar restart-timing configuration in Supervisor 🤷‍♂️
    – Eddie
    Commented Nov 3, 2023 at 11:39

2 Answers 2

2

What about the unattended-upgrades?

We had a similar problem with daemons (and still do) but not with the database. In our case, whenever an update occurs, with some chance daemons die with the same error(FATAL Exited too quickly). The supervisor is restarted, but somehow the daemon doesn’t seem to work.

To test this theory, look at when specific service (supervisor, mysql etc) crashed and compare it with the unattended-upgrade.log(everything should be in the /var/log/)

E.g. https://www.reddit.com/r/docker/comments/k4kk97/docker_daemon_crash_with_unattendedupgrades/

1
  • 2
    We're experiencing the same issue as OP, and Forge's auto updates (which I believe use unattended-upgrades) has definitely been floated as a possible culprit. I'm going to look closer into that; though if this is somehow related I still find it strange that the problem appears to exclusively occur on boxes w/ MySQL 8 (we haven't seen it occur on any of our MariaDB boxes to date, of which we have many). Commented Nov 5, 2023 at 22:17
2
+500

We're also seeing this issue on Laravel Forge. For us it mostly happens after a Craft update is deployed, so it might have something to do with the queue runner still having the previous Craft version loaded.

While I'm not sure what causes it, as a workaround you can ensure that the queue listener restarts periodically and/or after deployments.

There are some problems with this approach. For sites with a lot of traffic and queue jobs, restarting the queue runner may result in the queue getting backed up. Also, when a queue job is running while the queue runner is being forcefully restarted, it can cause the job to fail. To address this, we also have a nightly cron job that runs queue/retry to automatically retry failed queue jobs. But for small and medium sites that don't have tons of queue jobs coming in 24/7, this approach works quite well.

Restarting the queue runner periodically

Create a cron job in Forge, in the server settings under Scheduler:

  • Command: sudo supervisorctl restart all
  • User: forge
  • Frequency: Nightly (or choose whatever you like)

On newly provisioned servers, this should work without any further adjustments, because Forge now allows the forge user to execute supervisorctl commands without a password. On older servers, make sure the following snippet is in your /etc/sudoers.d/supervisorctl:

forge ALL=NOPASSWD: /usr/bin/supervisorctl reload
forge ALL=NOPASSWD: /usr/bin/supervisorctl reread
forge ALL=NOPASSWD: /usr/bin/supervisorctl restart *
forge ALL=NOPASSWD: /usr/bin/supervisorctl start *
forge ALL=NOPASSWD: /usr/bin/supervisorctl status *
forge ALL=NOPASSWD: /usr/bin/supervisorctl status
forge ALL=NOPASSWD: /usr/bin/supervisorctl stop *
forge ALL=NOPASSWD: /usr/bin/supervisorctl update *
forge ALL=NOPASSWD: /usr/bin/supervisorctl update

Restarting the queue runner after deployments

To restart the queue runner after every deployment, add this to the end of your deployment script:

sudo supervisorctl restart all

This will restart all daemons. Alternatively, you can also find the ID of your queue runner (use sudo supervisorctl status to list all active daemons) and only restart that in your deployment (replace <daemon-id> with the actual ID):

sudo supervisorctl restart <daemon-id>

On newer servers, if you're not using website isolation, this should work by default. If you are using website isolation, you need to allow the website user to run supervisorctl without a password, since you can't (or rather, shouldn't) provide the password in the deployment script.

Add this to your /etc/sudoers.d/supervisorctl:

websiteuser ALL=NOPASSWD: /usr/bin/supervisorctl *

Replace websiteuser with the website's (Ubuntu) user in Forge.

4
  • Thanks Moritz for a good and thoughtful answer. Unfortunately in my case, deployments doesn't seem to be connected with the issue. It has happened to pages that have been without deployments for weeks, and suddenly the daemon fails. The tips are still very useful though. Maybe regular restarting is necessary and will help 🤷‍♂️
    – Eddie
    Commented Nov 8, 2023 at 14:14
  • 1
    @Eddie Yeah some people on Discord said the same. In this case I would try the nightly restart. Of course, it's only a workaround, but this should work well enough for sites without huge amounts of traffic, so those won't pile up queue jobs while nobody is looking.
    – MoritzLost
    Commented Nov 8, 2023 at 14:28
  • 2
    Although this doesn't technically answer OPs question, I awarded the bounty to @MoritzLost as their answer at least offers some well-explained, concrete actions to mitigate the problem in the interim. Commented Nov 11, 2023 at 23:53
  • 1
    @MatsMikkelRummelhoff Yeah, unfortunate that for now there's only a workaround – hopefully someone can figure out why this is happening in the first place! Might also be something the Craft team needs to address, not sure if it's an infrastructure problem or a bug in the queue handler.
    – MoritzLost
    Commented Nov 13, 2023 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.