Security test generated large compiled_templates directory, site outage

Question

We just moved an installation of one of our sites to a new server. Previously, it was on a machine where the versions of Ubuntu / PHP prevented us from upgrading the Craft core version from 2.3.x to 2.6.x.

A few days ago, our client ran one of their security penetration tests that result in a large number of 404 HTTP requests against our site. This test generated enough unique cached templates in /craft/storage/runtime/compiled_templates to fill up the remaining disk space on the server - which appeared to take MySQL service down and serve 500 template errors afterwards.

Since bringing the site back up, we've been trying to identify and resolve the root issue behind the template caching that took the site offline. We've read this thread and are trying to formulate the right questions to prevent this from happening to this site the next time the client runs their tests.

1. Were there any significant changes in how Craft was handling template caching between versions 2.3 and 2.6 that would've allowed the old site to handle a spike in traffic without this issue?
2. Is there a configuration setting we missed to either avoid caching 404 pages, or reference the same cached template for all 404 requests?
3. Are there any special directory or file permission considerations to take into account that would prevent Craft from periodically clearing disk space here? Are there any automated cleanup tasks that should have run before the directory filled up?
4. Are there any OS, PHP or Apache settings that we should compare between the old server and new server that would help clarify the issue?

Answer 1

I don't have specific answers to your questions, but that's mainly because I feel there are some clarifications and observations that I think might be more helpful.

The thread you linked to is referring to template caching using the Craft {% cache %} tag in a template. All of its cached data is stored in the database in the craft_templatecache* tables, not on the file system. See here for an explanation of the different caching methods Craft uses.

The compiled files in craft/storage/runtime/compiled_templates are what happens when Twig needs to render a template in the craft/templates folder. The first time a template is requested, Twig will take that uncompiled template and compile it down to PHP and save it in the compiled_templates folder. If another request comes in for that same template, it will check the file's last modified date and if it hasn't changed, it will execute the compiled template directly.

Given that an actual 404 request wouldn't create new files in the compiled_templates folder (other than the 404 template that it might need to render), I'd guess that the assumption that the 404 requests caused the compiled_templates folder to fill up enough to run out of disk space is incorrect. It could be that enough log files or some other by-product of the testing caused more disk space to be used, but it's hard to tell without looking at the server.

You also mentioned that the database went down. Given that the {% cache %} tag stores its data in the database, my first hunch would be to look at the strategy for using the template cache tag on your templates. Here is a good article that weighs the in-and-outs of using it: https://nystudio107.com/blog/the-craft-cache-tag-in-depth

Two other things worth thinking about. A responsible security pen-tester is going to know better than to test on a live site by hammering it to the ground. But it's possible that maybe they were being responsible and the server is under-powered to handle any spike in load.