Import Large Number of records - 300,000

Question

Is there a best way to import a large number of records - About 315000 to be precise. The company provides a service which is only available to specific addresses. So I had set up a section to contain the addresses where the service is available. With fields - Address1, Address2, Postcode, Town Just got the data to import and discovered that there is 315000 records - I got the data in csv format. Tried with Import and FeedMe Plugins but run into memory issues straight away. CSV file is nearly 15MB and when converted to XML for feedme it is 52MB

So what are my options for importing?

Could I ramp up the resources on a Digital Ocean Server for the time that the import is required?

Or should I try another method to try this lookup service on the website?

Answer 1

Increasing your servers available memory and execution time can help increase the number of items you can import in a single file, but with the number of records you're dealing with it's likely, you'll need to break your import into multiple steps.

Consider breaking up the import files into smaller batches that can be imported within your environment's memory limit. As a first step, you could manually test for a limit of the max number of entries you can import.

This could be done by going back to the source that provided you the large CSV file and asking if they can provide you several CSV files of 50,000 records each, 10,0000 records each, or whatever number of records you found to be your limit. You may end up with lots of files and that's fine.

You could potentially also write a script that takes your CSV file or converted XML file and breaks it into several files on your own.

I'm not familiar with all the options in the Import and FeedMe plugins. My firm develops Sprout Import and we support batch importing multiple files. Sprout Import also requires data to be in a specific JSON format, which would likely require writing an additional script in your case as well.