Generating pending image transforms stuck

Question

My image transform task always gets stuck halfway through so that I have to "repair" the task as per this guide. Im trying to convert about 150 images (amounts to ~300 steps). Resetting the task within the database does work but it's not a permanent solution.

I've upped the php memory limit to 1024M, execution time is 0 and Apaches timeout is set to a ridiculously high number. Disk space is available and it's happening with imagemagick as well as gd. It also happens both on my local dev VM running Ubuntu 14.04 as well as on my Debian server. PHP and Apache error logs are empty on both machines. I'm running the Personal edition and the latest version of Craft.

I also tried to set ini_set('max_execution_time', 0); in craft/app/services/AssetTransformsService.php but it didn't change anything.

Some more info:

• Some image transforms are missing after restarting the task until it's complete. That renders the workaround by resetting the task in the db basically useless because not all images are transformed by the time it's complete.
• It's not a specific image that is preventing the task from completing, as it's always a different image that it's stuck on.
• There are usually 75-85 images converted until it's stuck.
• Update Asset Indexes doesn't help.
• Clearing all caches makes Craft convert all images again. It doesn't delete the old files, it just overwrites them (and gets stuck, as usual).

Answer 1

I was having this issue (w/ PHP 5.6, Craft 2.6.x, on WebFaction hosting) because it seemed like PHP had no memory limit and single processes were handling huge image transform queues, bombing when it got over 1.5gb or so.

Making sure Imagick has a memory cap seems to do the trick to avoid the queue getting stuck as PHP runs out of memory, which is discussed here:

Possible memory leak - GeneratePendingTransforms/ImageMagick

Relevant code:

// From http://stackoverflow.com/a/12835966
// Pixel cache max size
$this->setResourceLimit(\Imagick::RESOURCETYPE_MEMORY, 256);
// Maximum amount of memory map to allocate for the pixel cache
$this->setResourceLimit(\Imagick::RESOURCETYPE_MAP, 256);

in craft/app/vendor/pixelandtonic/imagine/lib/Imagine/Imagick/Imagick.php at line 33, at top of smartResize() function.

Update:

If you are not on shared hosting but on a VPS (or otherwise have sudo access on your server, OR are able install ImageMagick + php-imagick from source), you can set memory caps for ImageMagick directly using a policy.xml file, e.g.:

<policy domain="resource" name="memory" value="1GiB"/>
<policy domain="resource" name="map" value="1GiB"/>
<policy domain="resource" name="area" value="1GB"/>
<policy domain="resource" name="file" value="768"/>