0

I have a shopping product feed that is generated every day with around a million products. This can be in any format needed - CSV, JSON, XML etc, and will live on an S3 bucket.

Each day, I need to import this feed to entries in Craft and assign them categories etc. A large percentage of the products will be duplicates but it may need to update data within the product.

What would be the best way to import this large of a data set?

I have used FeedMe in the past, but I'm not sure whether it could handle this many products.

Thanks in advance.

Improve this question
2
  • 1
    Hey @fxfuture, do you have any further questions how to implement my suggestion? – Johannes May 28 at 11:01
  • @Johannes thank you for your suggestion. Would that be done as a custom module/plugin in Craft? Do you know of any documentation/tutorials that might help me do this? – fxfuture May 31 at 21:06
3

You could write a service method that reads the import file and divides it into chunks, then it could dispatch a queue job for each chunk which does the actual entry creation/update. This way you could scale the resources and amount of queue workers available to work through the queue and even Play with the chunk size.

In the queue job you might want to implement proper exception handling to keep track of unsuccessful entry operations to handle them somehow.

Improve this answer
5
  • 3
    In addition to this good advice, if as you say a large percentage will be duplicates (I assume this refers to products that already exist in Craft but have not been altered in the import source and therefore do not require re-importing), I'd recommend storing a hash/checksum in a custom field against each product so that you know whether you can skip the processing on the next import - this will save you lots of processing power and time taken to complete the import. I do a monthly import of around 60k products into Craft Commerce and this approach saves a significant amount of time. – James Smith May 23 at 10:17
  • @Johannes thank you for your suggestion. Would that be done as a custom module/plugin in Craft? Do you know of any documentation/tutorials that might help me do this? – fxfuture May 31 at 21:05
  • @fxfuture There is a whole area for extending Craft in the docs: craftcms.com/docs/3.x/extend/#modules-vs-plugins . Also, you should have a look at pluginfactory.io to generate a module scaffolding. You may structure your code in a Service class that utilizes an import Task. The service could be triggered via a Controller (HTTP) or a Console command. Regarding the hashing, you may have a look at Yii's concept of behaviors. – Johannes May 31 at 21:17
  • If you want to trigger the import Service on a regular basis I would suggest creating a thin Console command. This way, you could create a Cron job that runs the command whenever you want. – Johannes May 31 at 21:19
  • @Johannes that's really helpful thanks. I will take a look. And do you think the performance will be able to handle this many entries efficiently? – fxfuture May 31 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.