3

I am writing a plugin that imports a large amount of data form an API into some custom element types. I have it all set up and working nicely with Tasks however I am running into a fairly big issue with memory.

The plugin runs in the following way:

  1. Controlling Task gets set off.
  2. It sets off a SubTask twice that gets two different sets of data from the API and saves it to disk.
  3. It then sets off another bunch of SubTasks that import that data into my custom element types one element at a time - the number of Tasks at this point depends on how large the saved API data is as I split it into chunks of 100.

In my testing I am skipping point 2 from above, so it is only reading stuff from disk, importing it and repeating. I am finding that no matter what I do I can't stop the memory usage building up. I have used unset() in various places which has helped a little bit and have turned off the cacheElementQueries config parameter using craft()->config->set('cacheElementQueries', false); which has also helped but I'm still running into problems.

Some numbers

  • Resulting files from the API are 2.5MB and 5.3MB of json.
  • Total elements to import from each are 1200 and 2000 respectively.
  • As mentioned above I'm splitting the sets of data in to chunks of 100 to try and conserve memory.
  • php.ini memory_limit setting is 512M

Conclusion

I feel like I'm missing something obvious as php doesn't dump the memory until all the Tasks have run, or at least that is what it seems like. I kind of thought that after each Task or SubTask had completed the memory for that would be dumped.

I have to confess I'm still a bit new to serious programming and am only using htop to monitor the process and can see it going up continually until it reaches my php.ini memory_limit setting. I know I should probably use xdebug or something but that kind of scares me...

I've also read this: Memory issue when updating all entries from a channel (allowed memory size of X bytes exhausted (tried to allocate Y bytes))

If anyone has any ideas however wacky to help with plugging my leaky ship I'd be most grateful! I can post up code samples if it will help but I didn't want to make this post too lengthy.


Update

Here is a cut-down set of functions that are used in the process, I've not included lots of little bits that are either just doing processing on non-large variables or other mundane tasks.

Main Task

public function runStep($step)
{

    // Sync the stored API data to the Craft elements
    return $this->runSubTask('MyPlugin_Sync', null, array(
        'apiElement'         => $this->_apiElements[$step]['apiElement'],
        'apiElementSettings' => $this->_apiElements[$step]['apiElementSettings'],
        'apiService'         => $service,
        'offset'             => $this->_apiElements[$step]['offset']
    ));

}

Sync Task

public function getTotalSteps()
{

    $this->_apiData = craft()->myPlugin_ApiService->getElementsFromFile($this->_apiElement, $this->_offset, 100);

    // Check we got something
    if ( is_array($this->_apiData) )
    {
        return count($this->_apiData);
    }
    else
    {
        return 0;
    }

}

public function runStep($step)
{

    return craft()->myPlugin_ImportService->importElement($this->_apiService, $this->_apiElementSettings, array_shift($this->_apiData));

}

API Service

public function getElementsFromFile($apiElement, $offset = 0, $limit = null)
{

    // Get the file
    $filename = $this->_storage . $apiElement . 's.json';

    $file = IOHelper::getFile($filename);

    $data = json_decode($file->getContents(), true);

    // Try and help the memory
    unset($file);

    // Now select the right portion from it using our offset amount
    return array_slice($data, $offset, $limit);

}

public function mapData($data, $settings)
{

    $return = array();

    // Boring mapping code using $data

    unset($data);

    return $return;

}

Import Service

public function importElement($apiService, $apiElementSettings, $apiData)
{

    // using the API service, map the data to our fields/attributes held in the plugin settings
    $mappedData = craft()->myPlugin_ApiService->mapData($apiData, $apiElementSettings);

    // We’re done with this now, save our memory
    unset($apiData);

    // Standard code for getting an existing element and updating it
    // or creating a new one if none exist

    // Save the thing
    return craft()->myPlugin_myElement->saveMyElement($element);

}
  • If you wouldn't mind posting some code, it might help us see if there is something that is amiss. – Chad W Feb 12 '15 at 16:04
  • Also, I don't know if you've tried working with the phpMaxMemoryLimit config setting, although that might just be a bandaid in this case... buildwithcraft.com/docs/config-settings#phpMaxMemoryLimit – Chad W Feb 12 '15 at 16:06
  • Thanks @ChadW - yes I've set that config setting to the same as my php.ini one and I will try and get some pared down code samples up tonight / tomorrow. Thanks for taking a look! – Josh Angell Feb 12 '15 at 16:21
  • Hey @ChadW I've just put some code up there - again thanks for any help you can offer. – Josh Angell Feb 13 '15 at 8:48
  • I'll dig into this more later, but one thing that stands out is unset() doesn't necessarily free up memory, it just frees up the variable, as I understand it anyway. Thanks for posting your code. – Chad W Feb 13 '15 at 13:21
6

OK, finally answering my own question here as it seems to not be an issue with the latest Craft 2.5 builds when devMode is turned off, but is still an issue when devMode is on.

As far as I can tell its something to do with the Craft logging tool spiralling out of control, but thats as far as I got.

  • Just to say was having similar issues until disabling dev mode – Cole Henley Dec 15 '15 at 11:38
  • Yeah took me ages to work out! – Josh Angell Dec 15 '15 at 11:39
  • Also want to echo these guys, disabling dev mode fixed my memory issue with my tasks plugin. @JoshAngell have you contacted P&T about this? – Chase Giunta Dec 28 '15 at 19:03
  • @ChaseGiunta yeah I have an open ticket with Brad, I think as its not likely to creep into production (or shouldn't rather) and its a nightmare to test against it's not going to be sorted any time soon. I had to profile it to find it. – Josh Angell Dec 28 '15 at 19:14
  • craftcms.com/support/dev-mode This should cast some light on dev mode. Explains what is being logged and more – Krzysztof Boduch Nov 3 '16 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.