1

I build a plugin for Craft 2 which imports data from an excel file into craft commerce products.

The problem is that it needs to import excels with like 10.0000 rows and 30 columns.

I made a simple page in the CP where they can upload the file and then view the progress inside of the tasks.

Running just 1 task was not good because if it crashes on the first row it just stops. I wanted to make some kind of batching with tasks so this s what I came up with:

public function startExcelImportTasks($fileName){
    $return = false;
    $dataObject = craft()->myplugin_productsInfo->getDataByExcelName($fileName);
    $grandTotal = count((array)$dataObject);
    $batchSize = 500;
    if ($grandTotal > 0) {
        for ($i=0; $i < $grandTotal; $i+=$batchSize) { 
            $numberOfTasks = $batchSize;
            //if last loop set numberoftasks to total left
            if (($i + $batchSize) > $grandTotal){
                $numberOfTasks = $grandTotal - $i;
            }
            unset($dataObjectSlice, $dataObjectSliceString, $task);
            $dataObjectSlice = array_slice( $dataObject , $i , $numberOfTasks );
            $dataObjectSliceString = json_encode($dataObjectSlice);
            //create a task per batch
            $task = craft()->tasks->createTask('Myplugin_ImportExcel', 'Import excel - ('. $i . '-' . ($numberOfTasks + $i) .')', ['totalSteps' => $numberOfTasks, 'offset' => $i, 'grandTotal' => $grandTotal, 'dataObject' => $dataObjectSliceString]);
        }
        $return = true;
    }
    unset($dataObject,$grandTotal,$batchSize,$numberOfTasks,$dataObjectSlice,$dataObjectSliceString, $task);
    return $return;
}

So this is all running very well, I get tasks split up by 500 rows. My problem is memory usage, I thought when a new task would start it would free up the memory of the previous task but when I log my memory usage with memory_get_usage(true) it does not reset after the first loop and just continues to build up loop after loop until it finally crashes. It then goes to the next step starting with all free memory.

Is there any way to reset the memory for each loop? Loop 2 does not need to know ANYTHING that loop 1 needs since I slice the data and pass it to the right task.

I tried using some unsets in the code but that doesn't seem to do very much.

I also came across some subtask posts but I do not think that will change anything? I also like that the batching is shown in different tasks in the CP.

Any help is welcome,

Thanks

  • I since tried using a main task with batch sub tasks to run my import but the memory keeps adding up the more subtasks that have already ran (same result as above). – Pieter-Jan De Bruyne Jan 29 at 10:27
1

Ok, update..

The problem of my high memory usage was caused by (I think the logging of) devmode.. When I set devmode to false the moment that hit 10+gig memory before now has 100mb of memory usage...

Resetting memory on the different tasks that get initiated in the same function or as subtasks is not possible I guess?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.