I have a shopping product feed that is generated every day with around a million products. This can be in any format needed - CSV, JSON, XML etc, and will live on an S3 bucket.
Each day, I need to import this feed to entries in Craft and assign them categories etc. A large percentage of the products will be duplicates but it may need to update data within the product.
What would be the best way to import this large of a data set?
I have used FeedMe in the past, but I'm not sure whether it could handle this many products.
Thanks in advance.