Our application seems to be crashing periodically, on a seemingly unknown cadence. Sometimes it goes down once every few days, and sometimes (like today, when I'm at the end of my rope) it goes down 30 times a day or so.
When hitting the site on the frontend, the user gets a message saying that the site is unavailable; the maintenance mode error message. When we visit the admin panel, we are prompted to update two of our plugins, Geo and Embedder. After performing the updates the site comes back online.
In the craft logs before the site goes down, we're hit with a bunch of 404 errors:
Level error
Category exception.Craft\HttpException.404
Message Craft\HttpException in /var/www/vhosts/production/craft/app/controllers/TemplatesController.php:72
these range from requests trying to hit /wp-admin and other random pages
Then it looks like the site goes down, because then we get hit with 503s:
Level error
Category exception.Craft\HttpException.503
Message Craft\HttpException in /var/www/vhosts/production/craft/app/etc/web/WebApp.php:971
When we hit /admin, the logs show the following:
******************************************************************************************************
2018/06/02 20:52:43 [info] [application] [Forced] Preparing to update embedder.
2018/06/02 20:52:43 [info] [application] [Forced] Finished preparing to update embedder.
******************************************************************************************************
2018/06/02 20:52:43 [info] [application] [Forced] Starting to update the database.
2018/06/02 20:52:43 [info] [application] [Forced] The plugin "Embedder" wants to update the database.
2018/06/02 20:52:43 [info] [application] [Forced] Running migrations...
2018/06/02 20:52:43 [info] [application] [Forced] No new migration(s) found for the plugin Embedder. Your system is up-to-date.
2018/06/02 20:52:43 [info] [application] [Forced] Taking the site out of maintenance mode.
2018/06/02 20:52:43 [info] [application] [Forced] The plugin "Embedder" is done updating the database.
******************************************************************************************************
2018/06/02 20:52:43 [info] [application] [Forced] Starting to clean up after the update.
2018/06/02 20:52:43 [info] [application] [Forced] Clearing the update cache.
2018/06/02 20:52:43 [info] [application] [Forced] Flushing update info from cache.
2018/06/02 20:52:43 [info] [application] [Forced] Finished Updater.
2018/06/02 20:52:43 [info] [application] [Forced] Finished cleaning up after the update.
Now I don't think this is related to those two plugins; I think either Craft is getting out of sync somehow, or something is up with our cloud database.
Has anyone run into this issue before? What would be forcing the plugin migrations to initiate again and again, to make updates to the same database?