I build a plugin for Craft 2 which imports data from an excel file into craft commerce products.
The problem is that it needs to import excels with like 10.0000 rows and 30 columns.
I made a simple page in the CP where they can upload the file and then view the progress inside of the tasks.
Running just 1 task was not good because if it crashes on the first row it just stops. I wanted to make some kind of batching with tasks so this s what I came up with:
public function startExcelImportTasks($fileName){
$return = false;
$dataObject = craft()->myplugin_productsInfo->getDataByExcelName($fileName);
$grandTotal = count((array)$dataObject);
$batchSize = 500;
if ($grandTotal > 0) {
for ($i=0; $i < $grandTotal; $i+=$batchSize) {
$numberOfTasks = $batchSize;
//if last loop set numberoftasks to total left
if (($i + $batchSize) > $grandTotal){
$numberOfTasks = $grandTotal - $i;
}
unset($dataObjectSlice, $dataObjectSliceString, $task);
$dataObjectSlice = array_slice( $dataObject , $i , $numberOfTasks );
$dataObjectSliceString = json_encode($dataObjectSlice);
//create a task per batch
$task = craft()->tasks->createTask('Myplugin_ImportExcel', 'Import excel - ('. $i . '-' . ($numberOfTasks + $i) .')', ['totalSteps' => $numberOfTasks, 'offset' => $i, 'grandTotal' => $grandTotal, 'dataObject' => $dataObjectSliceString]);
}
$return = true;
}
unset($dataObject,$grandTotal,$batchSize,$numberOfTasks,$dataObjectSlice,$dataObjectSliceString, $task);
return $return;
}
So this is all running very well, I get tasks split up by 500 rows. My problem is memory usage, I thought when a new task would start it would free up the memory of the previous task but when I log my memory usage with memory_get_usage(true) it does not reset after the first loop and just continues to build up loop after loop until it finally crashes. It then goes to the next step starting with all free memory.
Is there any way to reset the memory for each loop? Loop 2 does not need to know ANYTHING that loop 1 needs since I slice the data and pass it to the right task.
I tried using some unsets in the code but that doesn't seem to do very much.
I also came across some subtask posts but I do not think that will change anything? I also like that the batching is shown in different tasks in the CP.
Any help is welcome,
Thanks
