I am writing a plugin that imports a large amount of data form an API into some custom element types. I have it all set up and working nicely with Tasks however I am running into a fairly big issue with memory.

The plugin runs in the following way:

  1. Controlling Task gets set off.
  2. It sets off a SubTask twice that gets two different sets of data from the API and saves it to disk.
  3. It then sets off another bunch of SubTasks that import that data into my custom element types one element at a time - the number of Tasks at this point depends on how large the saved API data is as I split it into chunks of 100.

In my testing I am skipping point 2 from above, so it is only reading stuff from disk, importing it and repeating. I am finding that no matter what I do I can't stop the memory usage building up. I have used unset() in various places which has helped a little bit and have turned off the cacheElementQueries config parameter using craft()->config->set('cacheElementQueries', false); which has also helped but I'm still running into problems.

Some numbers

  • Resulting files from the API are 2.5MB and 5.3MB of json.
  • Total elements to import from each are 1200 and 2000 respectively.
  • As mentioned above I'm splitting the sets of data in to chunks of 100 to try and conserve memory.
  • php.ini memory_limit setting is 512M


I feel like I'm missing something obvious as php doesn't dump the memory until all the Tasks have run, or at least that is what it seems like. I kind of thought that after each Task or SubTask had completed the memory for that would be dumped.

I have to confess I'm still a bit new to serious programming and am only using htop to monitor the process and can see it going up continually until it reaches my php.ini memory_limit setting. I know I should probably use xdebug or something but that kind of scares me...

I've also read this: Memory issue when updating all entries from a channel (allowed memory size of X bytes exhausted (tried to allocate Y bytes))

If anyone has any ideas however wacky to help with plugging my leaky ship I'd be most grateful! I can post up code samples if it will help but I didn't want to make this post too lengthy.


Here is a cut-down set of functions that are used in the process, I've not included lots of little bits that are either just doing processing on non-large variables or other mundane tasks.

Main Task

public function runStep($step)

    // Sync the stored API data to the Craft elements
    return $this->runSubTask('MyPlugin_Sync', null, array(
        'apiElement'         => $this->_apiElements[$step]['apiElement'],
        'apiElementSettings' => $this->_apiElements[$step]['apiElementSettings'],
        'apiService'         => $service,
        'offset'             => $this->_apiElements[$step]['offset']


Sync Task

public function getTotalSteps()

    $this->_apiData = craft()->myPlugin_ApiService->getElementsFromFile($this->_apiElement, $this->_offset, 100);

    // Check we got something
    if ( is_array($this->_apiData) )
        return count($this->_apiData);
        return 0;


public function runStep($step)

    return craft()->myPlugin_ImportService->importElement($this->_apiService, $this->_apiElementSettings, array_shift($this->_apiData));


API Service

public function getElementsFromFile($apiElement, $offset = 0, $limit = null)

    // Get the file
    $filename = $this->_storage . $apiElement . 's.json';

    $file = IOHelper::getFile($filename);

    $data = json_decode($file->getContents(), true);

    // Try and help the memory

    // Now select the right portion from it using our offset amount
    return array_slice($data, $offset, $limit);


public function mapData($data, $settings)

    $return = array();

    // Boring mapping code using $data


    return $return;


Import Service

public function importElement($apiService, $apiElementSettings, $apiData)

    // using the API service, map the data to our fields/attributes held in the plugin settings
    $mappedData = craft()->myPlugin_ApiService->mapData($apiData, $apiElementSettings);

    // We’re done with this now, save our memory

    // Standard code for getting an existing element and updating it
    // or creating a new one if none exist

    // Save the thing
    return craft()->myPlugin_myElement->saveMyElement($element);

  • If you wouldn't mind posting some code, it might help us see if there is something that is amiss.
    – Chad W
    Commented Feb 12, 2015 at 16:04
  • Also, I don't know if you've tried working with the phpMaxMemoryLimit config setting, although that might just be a bandaid in this case... buildwithcraft.com/docs/config-settings#phpMaxMemoryLimit
    – Chad W
    Commented Feb 12, 2015 at 16:06
  • Thanks @ChadW - yes I've set that config setting to the same as my php.ini one and I will try and get some pared down code samples up tonight / tomorrow. Thanks for taking a look! Commented Feb 12, 2015 at 16:21
  • Hey @ChadW I've just put some code up there - again thanks for any help you can offer. Commented Feb 13, 2015 at 8:48
  • I'll dig into this more later, but one thing that stands out is unset() doesn't necessarily free up memory, it just frees up the variable, as I understand it anyway. Thanks for posting your code.
    – Chad W
    Commented Feb 13, 2015 at 13:21

1 Answer 1


OK, finally answering my own question here as it seems to not be an issue with the latest Craft 2.5 builds when devMode is turned off, but is still an issue when devMode is on.

As far as I can tell its something to do with the Craft logging tool spiralling out of control, but thats as far as I got.

  • Just to say was having similar issues until disabling dev mode Commented Dec 15, 2015 at 11:38
  • Yeah took me ages to work out! Commented Dec 15, 2015 at 11:39
  • Also want to echo these guys, disabling dev mode fixed my memory issue with my tasks plugin. @JoshAngell have you contacted P&T about this? Commented Dec 28, 2015 at 19:03
  • @ChaseGiunta yeah I have an open ticket with Brad, I think as its not likely to creep into production (or shouldn't rather) and its a nightmare to test against it's not going to be sorted any time soon. I had to profile it to find it. Commented Dec 28, 2015 at 19:14
  • craftcms.com/support/dev-mode This should cast some light on dev mode. Explains what is being logged and more Commented Nov 3, 2016 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.