Cache issues making site super slow in production but not in dev mode

Question

We have a site that is performing very poorly with very long load times and I'm guessing it may be related to caching but have no idea how/why.

The site's homepage was loading very slowly while in development due to a mega menu with a lot of queries to create primary and secondary navigation. I cached that globally while in dev and the page performance improved about 70%. I did some further global caching {% cache globally %} for things like the site footer and than added some regular caching {% cache %} for entry-specific template content.

Once this was all done, performance was very good. However, then we launched the site and took the site out of dev mode (which is had previously been in). The launch was on the same account - nothing changed - just pointed the DNS to this account.

Since that time, performance has been poor again. Further, when logging into the control panel, we always see this (which we never saw while in dev mode):

What's odd also is the circle and bar progress indicators are always in the exactly pictured state - they're never more or less along in terms of position (over the course of weeks).

Lastly, if I go into the CP settings area and Clear Caches, performance seems to pick up again for a little while.

I'm at a complete loss as to what may be happening that is causing these issue and any help would be greatly appreciated.

Thanks, Mike

Answer 1

This article (http://buildwithcraft.com/help/stuck-tasks) from Brad's comment was indeed the way to address this - and our logs did show this exceeding the php execution time. Thanks Brad!

Answer 2

For others that might be facing an issue similar to this – we found that our Craft site was gradually getting slower and slower over a period of several days, despite using the {% cache %} tag. When we cleared the template cache, the site would be fast again until the cache grew in size and got slow again.

It was our uninformed use of the {% cache %} tag that contributed to this.

It's important to firstly understand that using {% cache %} without any parameters generates a separate copy of the block for every single URL that uses it. If the page is requested with a different querystring, again the block is regenerated and a new copy is saved to the cache. This means you could potentially have many dozens, hundreds, or even thousands of copies of the same block in your cache.

Because we had several search pages with URLs like /search?q=searchterm, this kept creating more and more cached blocks every time somebody did a search.

So in our case we had to be more explicit when using the cache tag, in two ways:

1. For any block that should always output the same, regardless of URL, use

   {% cache globally %}
   

2. For any block that does change based on URL, but perhaps not on querystring (such as perhaps a menu/nav block) then use something like:

   {% cache globally using key "navbar:" ~ craft.request.path %}
   

   (as mentioned in the Craft docs)

And here are the results:

Answer 3

This will probably help you: The Craft {% cache %} Tag In-Depth