3

I've got an asset folder where i need to bring over about 20Gb of assets, obviously trying to update asset indexes from the Craft Cpanel is causing a serious failure.

Other than adding my assets in smaller chunks is there any other option to make it less memory intensive?

If the assets were on S3 rather than local, would it be less intense?

5

I've done this before with a combination of a Craft console command that replicates the indexing process, and a external way that processes each row.

<?php
namespace Craft;

class BusinessLogicCommand extends BaseCommand
{
    /**
    * ./craft/app/etc/console/yiic businesslogic --offset=x --sourceId=y --sessionId=z
    */
    public function actionIndex ($offset = '', $sourceId = '', $sessionId = '')
    {
        if ( $offset ) {
            $index = craft()->assetIndexing->processIndexForSource($sessionId, $offset, $sourceId);

            if ( $index ) {
                echo sprintf("Processed offset: %s for source %s", $offset, $sourceId);

                return 1;
            }
        }

        return 0;
    }

    /**
    * ./craft/app/etc/console/yiic businesslogic getIndexOverview
    */
    public function actionGetIndexOverview ($sourceIds = '')
    {

        // Get batches
        $batches   = array();
        $sessionId = craft()->assetIndexing->getIndexingSessionId();

        // Selection of sources or all sources?
        if ( $sourceIds ) {
            $sourceIds = explode(',', $sourceIds);
        }
        else {
            $sourceIds = craft()->assetSources->getViewableSourceIds();
        }

        $missingFolders = array();
        $grandTotal     = 0;

        foreach ($sourceIds as $sourceId) {
            // Get the indexing list
            $indexList = craft()->assetIndexing->getIndexListForSource($sessionId, $sourceId);

            if ( !empty($indexList['error']) ) {
                echo json_encode($indexList);

                return 0;
            }

            if ( isset($indexList['missingFolders']) ) {
                $missingFolders += $indexList['missingFolders'];
            }

            $batch = array();

            for ($i = 0; $i < $indexList['total']; $i++) {
                $batch[] = [
                    'sessionId' => $sessionId,
                    'sourceId'  => $sourceId,
                    'total'     => $indexList['total'],
                    'offset'    => $i,
                    'process'   => 1,
                ];
            }

            $batches[] = $batch;
        }


        // Overview

        $assetsSourcesBeingIndexed = $sourceIds;
        $assetsMissingFolders      = $missingFolders;

        $job = [
            'batches' => $batches,
            'total'   => $grandTotal,
        ];

        echo json_encode($job);

    }


    public function actionCleanUpIndex ()
    {
        // Get overview
        $sourceIds      = $assetsSourcesBeingIndexed;
        $missingFiles   = craft()->assetIndexing->getMissingFiles($sourceIds, $sessionId);
        $missingFolders = $assetsMissingFolders;

        $responseArray = array();

        if ( !empty($missingFiles) || !empty($missingFolders) ) {
            $responseArray['confirm'] = craft()->templates->render('assets/_missing_items', array( 'missingFiles' => $missingFiles, 'missingFolders' => $missingFolders ));
            $responseArray['params']  = array( 'finish' => 1 );
        }
        // Clean up stale indexing data (all sessions that have all recordIds set)
        $sessionsInProgress = craft()->db->createCommand()
                                        ->select('sessionId')
                                        ->from('assetindexdata')
                                        ->where('recordId IS NULL')
                                        ->group('sessionId')
                                        ->queryScalar();

        if ( empty($sessionsInProgress) ) {
            craft()->db->createCommand()->delete('assetindexdata');
        }
        else {
            craft()->db->createCommand()->delete('assetindexdata', array( 'not in', 'sessionId', $sessionsInProgress ));
        }

    }

}

This is done in 3 steps:

  1. Trigger getIndexOverview, which returns an JSON object string that lists all your index rows.
  2. Use the JSON object in some way, and create a job for each row, which should trigger step #3.
  3. Trigger the index command with the parameters --offset=x --sourceId=y --sessionId=z

I have used Laravel + Queue backed by Redis for this. If you have huge files, or a lot of data to process, think about spinning up a big VPS somewhere.

That way you can run 10-12 jobs in parallel, export your database/file structure, and tear the VPS down after. With hourly pricing, you'd end up paying a few $.

3

I've encountered this issue myself when syncing asset indexes with S3 and after going through the Pixel & Tonic support channels, they advised me to create a plugin.

Currently my plugin only indexes sources that are held on S3 but I may update it in future to also do local sources. Assets will be getting a big overhaul for Craft 3 though. If you wish to move your assets to S3 and cannot find any other solutions to get it working then here is my plugin to help the indexing service: https://github.com/a-digital/assetindexer

I hope you manage to get it sorted.

  • Hey Matt, that looks great, i'll give that a go!! Thanks!! – John Dec 8 '16 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.