1

I'm currently implementing a task-based application, which pulls some data from an external API before creating a number of entries based on the payload.

After creating the entries, the app will need to download a fair amount of images, storing the files as Assets and adding them to an Assets fields on one of the entries (each entry has a set of ~30 related images on an external server, that needs to be pulled).

Currently, I'm running 1 task per entry, with a sub task for downloading all of the images related to each entry (1 step per image, so ~30 steps per sub task). The images are quite large – around 5MB each.

The sub task takes an entry ID and an array of image URLs and metadata as its settings. Each step of the sub task looks like this:

  1. The entry the image will be related to is pulled from the entry ID.
  2. The task attempts to pull any existing Asset for the image (criteria based on filename)
  3. If Asset not found, the remote image is downloaded using Guzzle to a temporary folder, before the Asset is created using craft()->assets->insertFileByLocalPath.
  4. Another criteria to pull the newly created Asset.
  5. Asset is updated with the metadata (which could've changed, regardless of whether the Asset was previously saved)
  6. The IDs for any Assets in the entries Asset field is pulled using $entry->assetField->ids().
  7. The new Asset's ID is added to the ID array, the asset field is updated with $entry->getContent()->setAttribute('assetField', $ids); and the entry is resaved through craft()->entries->saveEntry.
  8. The temporary image file is deleted using the IOHelper class.

This is running on a 2GB droplet, where PHP/Craft has access to the full 2GB RAM – which I assumed should be plenty, though the server is running out of memory, intermittently. At one point, MySQL even crashed and had to be restarted :/

The memory issues never appear when Craft is running one of the regular, parent tasks – it always happens when one of the image sub tasks is running, which is why I'm debating my current setup.

From Brad Bell's comment to this answer, I know that 1 task = 1 request, even if that task has multiple steps and/or sub tasks. What I take from that is that having loads of sub tasks is potentially more dangerous in terms of memory usage and possible timeout issues, than having loads of smaller regular tasks. Is this a correct assumption?

In closing, which of the following approaches would be the safest, in terms of memory usage (and/or other issues I might not have considered):

  • One regular task per image (~30 tasks)
  • One regular task per image set (~30 steps)
  • One sub task per image (~30 sub tasks)
  • One sub task per image set (~30 steps, my current implementation)

I should mention that in this case, it doesn't really matter if the parent task completes before the image tasks run/complete, so I don't actually need to use sub tasks for the downloads.

Also, devMode is currently false on the server and I'm still having issues.

  • Just double checking, but you've also bumped this up as well? craftcms.com/docs/config-settings#phpMaxMemoryLimit That's an additional layer on top of whatever your php.ini file says. – Brad Bell Mar 15 '16 at 23:20
  • 1
    @BradBell Yeah. Currently set to 2048M. I'm going to set up Blackfire and hopefully figure out where the memory goes :) – Mats Mikkel Rummelhoff Mar 15 '16 at 23:23
  • Yeah, hard to say. A lot of it is dependent on what your subtasks are actually doing. Blackfire is pretty slick for stuff like this. – Brad Bell Mar 15 '16 at 23:26
  • I'll add some detail on the sub tasks to my question – still curious about the basic issue, though; is it possible to say something general regarding resource consumption for lots of tasks vs. lots of sub tasks? – Mats Mikkel Rummelhoff Mar 15 '16 at 23:28
  • 1
    Probably worth mentioning you can affect some PHP garbage collection settings from php.ini that will affect memory usage, too. secure.php.net/manual/en/session.configuration.php – Brad Bell Mar 15 '16 at 23:48
2

I think this is difficult to answer properly because it depends almost entirely on what the tasks and subtasks are actually doing.

For example, the main task could do something that ate 400MB of memory and triggered 400 sub-tasks that did nothing but return null not using any memory.

Alternatively, the main task could do nothing but trigger subtasks (using very little memory itself) but each subtask could use 400MB of memory.

Obviously with subtasks you have to me more conscious of memory usage because usually there will be multiple subtasks and there is only ever one main task.

  • 1
    Thanks! That last paragraph is sort of what the wall of text was really aiming at. Which would be worse: A trillion sub tasks (all belonging to the same main task) consuming 400MB each, or a trillion main/regular/standalone tasks consuming 400MB each (with no sub tasks)? – Mats Mikkel Rummelhoff Mar 15 '16 at 23:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.