I'm currently implementing a task-based application, which pulls some data from an external API before creating a number of entries based on the payload.
After creating the entries, the app will need to download a fair amount of images, storing the files as Assets and adding them to an Assets fields on one of the entries (each entry has a set of ~30 related images on an external server, that needs to be pulled).
Currently, I'm running 1 task per entry, with a sub task for downloading all of the images related to each entry (1 step per image, so ~30 steps per sub task). The images are quite large – around 5MB each.
The sub task takes an entry ID and an array of image URLs and metadata as its settings. Each step of the sub task looks like this:
- The entry the image will be related to is pulled from the entry ID.
- The task attempts to pull any existing Asset for the image (criteria based on filename)
- If Asset not found, the remote image is downloaded using Guzzle to a temporary folder, before the Asset is created using
- Another criteria to pull the newly created Asset.
- Asset is updated with the metadata (which could've changed, regardless of whether the Asset was previously saved)
- The IDs for any Assets in the entries Asset field is pulled using
- The new Asset's ID is added to the ID array, the asset field is updated with
$entry->getContent()->setAttribute('assetField', $ids);and the entry is resaved through
- The temporary image file is deleted using the IOHelper class.
This is running on a 2GB droplet, where PHP/Craft has access to the full 2GB RAM – which I assumed should be plenty, though the server is running out of memory, intermittently. At one point, MySQL even crashed and had to be restarted :/
The memory issues never appear when Craft is running one of the regular, parent tasks – it always happens when one of the image sub tasks is running, which is why I'm debating my current setup.
From Brad Bell's comment to this answer, I know that 1 task = 1 request, even if that task has multiple steps and/or sub tasks. What I take from that is that having loads of sub tasks is potentially more dangerous in terms of memory usage and possible timeout issues, than having loads of smaller regular tasks. Is this a correct assumption?
In closing, which of the following approaches would be the safest, in terms of memory usage (and/or other issues I might not have considered):
- One regular task per image (~30 tasks)
- One regular task per image set (~30 steps)
- One sub task per image (~30 sub tasks)
- One sub task per image set (~30 steps, my current implementation)
I should mention that in this case, it doesn't really matter if the parent task completes before the image tasks run/complete, so I don't actually need to use sub tasks for the downloads.
devMode is currently
false on the server and I'm still having issues.