How to best design large queue jobs?

Question

I have a plugin that captures data and stores it as an Element, similar to a log. There can be a lot of Elements created so there is also a way to purge those Elements from the DB. In most cases, the amount of Elements being purged at a given time is reasonable. However, there are some scenarios where a user may be triggering the deleting of tens of thousands of elements.

What is the best way to handle this so that I'm least likely to crash a users site

I'm not sure what the right question to ask is so here are a few areas that a better understanding would be helpful:

1. Does creating a single job with 10,000 steps use more resources than creating 100 jobs with 100 steps?
2. If breaking things out into batches makes sense, what is a reasonable batch size for modifying Elements using a job?
3. Would the need to do different actions lead to a different answer? For example, would the same recommendations hold for modifying Elements and deleting Elements?

Answer 1

The answers to your questions will inevitably be, that it depends, however I have tried to present examples of situations that may help clarify things a bit further.

1. Does creating a single job with 10,000 steps use more resources than creating 100 jobs with 100 steps?

Each queue job is stored as a row in the queue database table. If a job had 10,000 steps and needed to store a unique piece of information about each step, then the amount of data stored in the job column (a LONGBLOB) could be very large. Fetching, storing that large blob in memory and working with it would require more resources than working with smaller blobs one at a time. If part of the job execution involved fetching and working with 10,000 elements, then again that would require more resources than working with smaller collections of elements. So in theory, creating multiple, smaller jobs would take longer to complete but would be less prone to the memory and/or time limit being exceeded and the job failing.

2. If breaking things out into batches makes sense, what is a reasonable batch size for modifying Elements using a job?

If you want to account for queue jobs that are run through the browser (in addition to through CLI commands) then you should try to avoid allowing a job to exceed the memory limit at all costs. However, since the memory limit can be different on every environment (Craft 3 requires at least 256MB of memory allocated to PHP) this can be difficult to do. 100 feels like a conservative batch size but again, it depends on how many resources the job requires.

What I have done in the Campaign plugin is added a block of code to each iteration in the job that checks whether the memory used by the current job (memory_get_usage(true)) and the time that has passed since it was requested (time() - $_SERVER['REQUEST_TIME']) are nearing the respective limits. If they are then I create a new job, give it a delay, add it to the queue and stop execution of the current job. That means that the SendoutJob can continue sending emails indefinitely and without a fixed limit, until it detects that the job is close to failing. Setting a delay on the new job allows the current request to end without failing and the new job will be executed the next time the queue is run. Here is the relevant code:

// If we're beyond the memory limit or time limit or max batch size has been reached
if (memory_get_usage(true) > $memoryLimit || time() - $_SERVER['REQUEST_TIME'] > $timeLimit || $count >= $batchSize) {
    // Add new job to queue with delay
    Craft::$app->getQueue()->delay($settings->batchJobDelay)->push(new self([
        'sendoutId' => $this->sendoutId,
        'title' => $this->title,
        'batch' => $this->batch + 1,
    ]));

    return;
}

Note that $memoryLimit and $timeLimit have been set to thresholds lower than that of the server (source).

3. Would the need to do different actions lead to a different answer? For example, would the same recommendations hold for modifying Elements and deleting Elements?

If you decide to set a fixed limit then it would make sense that it should probably be lower for resource intensive jobs. If the completion of the job is critical then you might consider implementing a similar approach to the Campaign plugin's SendoutJob, which is not perfect but has worked well in the field.

If this is a distributed plugin then the best you can do is try to avoid jobs failing and communicate to your users what they can do to avoid it, as described in Robust queue job handling in Craft CMS.

See also: How to prevent "Failed" errors in job scheduler in Craft CMS 3?

Answer 2

So if you're running the queue jobs via one of the techniques outlined in the Robust queue job handling in Craft CMS article, it shouldn't matter.

But I'm guessing you're doing this in the context of a plugin, where you can't control what setup the user has for executing their queue jobs.

In that case, I'd opt for breaking it up into smaller queue jobs that have a better chance of finishing before one of the various web-based timeouts happens. You'll also potentially benefit from concurrency, although it really depends on the CPU load and a number of other complex factors.

Mostly, I'd be interested in resiliency.