Import a large number of assets via feedme

Question

I have 224,559 assets (images) I need to import and assign to entries. Assets are currently stored in folders that match their original asset id. Imported assets will be stored on s3. I'm using feedme for the import so far.

I have json for both the assets and entries. A typical record for assets looks like:

{
    "title": "Ceiling decor at The Atomic Tiki in Memphis, TN",
    "import": {
        "key": "asset-1197-16772",
        "source": "TikiTime",
        "sourceUrl": "https://tikitime.com/images/locations/1197/16772_jumbo.JPG",
        "outputPath": "/Volumes/Sites/Personal/mytiki.life/web/uploads/master-assets/tiki-bars/1197/16772.jpg",
        "date": "2020-04-23T08:09:01+00:00"
    }
},

A typical entry json entry looks like:

{
    "title": "Sun Hung Heung Green Bucket",
    "originalId": 2,
    "assets": [
        {
            "title": "Sun Hung Heung Green Bucket - 2-153663",
            "key": "ooga-mooga-asset-2-153663",
            "outputPath": "/Volumes/Sites/Personal/mytiki.life/web/uploads/master-assets/tiki-mugs/2/153663.jpg"
        },
        {
            "title": "Sun Hung Heung Green Bucket - 2-153664",
            "key": "ooga-mooga-asset-2-153664",
            "outputPath": "/Volumes/Sites/Personal/mytiki.life/web/uploads/master-assets/tiki-mugs/2/153664.jpg"
        }
    ]
}

The import.sourceUrl property is the local file path of the file. All the images are broken up into folders that match their original entry ids.

I've setup a field in the entry channel for the asset relationship that's restricted to a single folder with a subdirectory pattern: {section.handle|kebab}/{id} -> tiki-mugs/875

Obviously the new (imported) entry id doesn't match the old entry id so… I could export all the new entry ids and then rename all the folders to match but that would be painful.

I'm having trouble figuring out how to relate the assets with the entries and get them in the correct directories.

I thought I could possibly leave the files in their current location and run the asset index command. That would create the records. Then somehow assign them in feedme. If I did that would the assets be moved when the entry gets saved?

Any other ideas?

Answer 1

I ended up writing my own command line importer rather than using feedme. This meant I could be much more flexible with my import.

I also had to break up the sub folders into groups given the Assets admin panel cannot deal with 1000's of sub folders. See: https://github.com/craftcms/cms/issues/2547#issuecomment-537934580