queue/listen occasionally loses access to the database:
FATAL Exited too quickly
Database Connection Error: Craft CMS can’t connect to the database.
As the connection is lost, it seems like the daemon is not able to recover, and needs to be manually restarted.
- Database: MySQL 8.0.35
- There was no deployment being done at the time of failure.
- I don’t currently have any automatic restarting of the daemon after deployments.
- There are no other indications of the database being down at the time the daemon dropped out.
- It has happened across multiple sites, and across hosts.
- One common factor is that Laravel Forge is used for provisioning and setting up the Daemon.
- It has happened to custom VPS’s, and Digital Ocean Droplets.
- All affected sites as far as I know are Craft 4.5.5 or later.
- This issue is new to me the last 3-5 months, even though I've made Craft sites with this same setup for a couple of years.
- The issue has also been discussed in the Discord channel "Devops" late August 2023. https://discord.com/channels/456442477667418113/456448961679589386/1144029528394580099
- It seems to have a tendency to happen at night time. Which is also roundabout the same time doing database backups. It doesn't correlate greatly, so it seems farfetched to be linked. Often linked by +- 2 hours.
Most of the time this happens, its quite critical to get up and running again, and I don't know how to really get to the bottom of the problem.
Daemon Supervisor Configuration
[program:daemon-758141] command=/usr/bin/nice -n 10 /usr/bin/php /home/forge/www.site.com/craft queue/listen --verbose process_name=%(program_name)s_%(process_num)02d autostart=true autorestart=true user=forge numprocs=2 startsecs=1 redirect_stderr=true stdout_logfile=/home/forge/.forge/daemon-758141.log stopwaitsecs=10 stopsignal=SIGTERM stopasgroup=true killasgroup=true