Painfully slow Feed Me import of User accounts

Question

I'm aware of this thread on Feed Me slow performance, but I'm raising it as a separate thread in case there is anything particular about my circumstance that is pertinent.

I'm running Craft 3.4.18 and trying to import a large number of users (about 1,500) into a specific user group using Feed Me:

1. I'm using .csv sources on the local file system above web root
2. I'm importing:
   • Username (as email address string)
   • Email
   • First name
   • Last name
   • Shared password
   • User group string
3. I'm batching imports in 500s but because things are so slow will now try smaller batches
4. I have read and attended to the performance aspects of the Feed Me troubleshooting docs
5. There is plenty of memory and execution time allocation to Craft and the instance is on a dedicated Linux server with 8Gb of RAM

Imports are working without error, but paaaaaainfully slowly -- we're talking hours -- to the point of the import process being effectively useless*.

I have looked into Andrew Welch's #1 solution to running the queue as a background process, but have bumped into installation issues with the Async Queue plugin.

Any suggestions much appreciated.

---

I'd be happy in principle to do SQL inserts directly into the craft_users and craft_usergroups_users tables but I infer that's not an option since each hash of the shared password is unique and there is a unique UID to generate for each row too.

---

Update on #3 above:

I'm currently running a batch of 100 instead of 500 and that makes no difference to speed whatsoever.

Next step is perhaps to try a JSON feed instead of CSV?

---

*'Effectively useless' in the sense that although I have time to run extended imports right now, if it's going to generally take this long it's a non-starter for time-pressured imports.

---

Second update on #3 above:

An import batch of 100 I started over 2 hours ago has crept along incrementally, with me periodically reviewing its status during that time and seeing barely any change, and then…

I have just revisited it and it suddenly went from 64% complete and unchanging to 100%. Now onto the next batch to see what happens.

Answer 1

Turns out the answer in my case is to use a JSON data source.

Importing users as .json instead of .csv appears to be about 10 times faster:

• With .csv importing 480 users took 2.5 hours
• With .json importing 455 users took 15 minutes

My JSON looked like this:

{"user": [
    {
        "username":"[email protected]",
        "email":"[email protected]",
        "firstName":"Jane",
        "lastName":"Doe",
        "password":"User pass phrase",
        "groupName":"User group name"
    },
    {
        "username":"[email protected]",
        "email":"[email protected]",
        "firstName":"Joe",
        "lastName":"Bloggs",
        "password":"User pass phrase",
        "groupName":"User group name"
    }
]}