Delete Stale Template Caches tasks much larger in Production than Staging

Question

I've been running into issues where the Delete Stale Template Caches tasks on my Production environment are running up to 20x longer than on my identical Staging environment. The config/environment settings are the same across both (including devMode being disabled on Stage) but they use separate databases. So I imported my Prod database into my Stage environment just to get complete parity across both environments to try and diagnose the problem.

I edited the same entry on Prod/Staging and hit save, both kick off the DSTC task, but I got ~10,000 steps on Production as opposed to ~500 on Staging.

I copied the SQL row on each database and this is what I get:

Stage:

14  14  1   2   0   263 555 running DeleteStaleTemplateCaches   NULL    {"elementId":["3282","3342","3343","3341","3548","3549","3550","3547","3382","3383","3381","9171","9172","3395","3391","14629","14630","14628","3281"]} 2016-07-14 18:38:46 2016-07-14 18:38:52 d39e2b0f-e6d6-446d-9d2b-30d35ecca378

Production:

16042   16042   1   2   0   439 9864    running DeleteStaleTemplateCaches   NULL    {"elementId":["3282","3342","3343","3341","3548","3549","3550","3547","3382","3383","3381","9171","9172","3395","3391","14629","14630","14628","3281"]} 2016-07-14 18:36:46 2016-07-14 18:37:07 b152c337-2316-45c9-8b29-4716faf41897

The settings of the tasks are the exact same as far as the ElementIds that it involves, but the Stage task is 555 steps while the Production task is 9864. I'm seeing this pretty consistently across all entries - the number of steps in the Production environment is around 20x the Staging on identical entries.

I'm a bit stumped as to why this is happening -- what determines the number of steps involved in any given DSTC task?

EDIT: Another odd thing is that it's kicking off the tasks on entries that don't have any cache tags inside the templates. Again this is only happening in Production, it seems that the Staging site seems to recognize that these templates shouldn't be getting cached in the first place.

Answer 1

First, this is perfectly normal. :)

Have a look at Brandons explanation here on how the cache tag works.

If the element was added or modified, it queues up a new “Delete Stale Template Caches” task, which cycles through the templatecachecriteria table to see if there are any cached element queries which (if not cached) would have started showing the just-saved element. If the query returns the element, its corresponding cache will be deleted.

So, the number of steps corresponds to the number of rows in the templatecachecriteria table in the database.

Why would this be different between staging and production? Probably because you're getting different kind of traffic on the production site than the staging site.

It's important to understand that the cache is tied to the url, so mydomain.com/news and mydomain.com/news?id=212 will result in two different caches being created in the database, and a number of rows in the templatecachecriteria table being created based on the number of criterias created inside the cached code (which can be alot).

The example given above is an example that we encounter a lot, where when an old site is replaced by a new one, there will still be a lot of links pointing to the new one. If these aren't handled correctly by either rewriting them to new url's, or returning 404's, this could result in many old urls generating new caches for the same page.

The recommended way to avoid this is to use the globally keyword, and using the request path as the key, see the example in the "using key" section of the docs for {% cache %}.

Another common caveat is that there are caches on the 404 page that is not declared as globally, for instance a menu that needs a selected state or something. Then all urls that results in a 404 not found page, will create new caches with new criterias in templatecachecriteria. I've seen examples where this will eventually just kill the server because of the number of criterias that are rerun when an entry is saved.

If I were you I'd have a look in the templatecaches table, and look at the path column to identify the paths of the created caches. This will often reveal groups of urls which generates more caches than they ideally should, and you could work from there to reduce the number.