2

I am wondering if I have structured my data badly and was hoping someone could advise on an alternative. Essentially I have a structure with two types, a catalogue and the catalogues children. Some catalogues may have up to 400 children.

When I began importing the data using Feed Me I didn't have any problem, but after a while it was becoming nearly impossible to use and I'm only half way through. I checked the logs in the debug bar and found this was holding things up. Perhaps it's of some help?

Any advice here would be really appreciated!

enter image description here

Edit:

For more clarification, this log is from the screen where you map the fields. There may be around 15,000 entries so far, most of which are the children entries. I am just wondering if there is something wrong with using structures this way or is this just because of the large amount of entries in the database? I have read that people are running Craft sites with over 100,000 entries so I didn't think this would be a problem.

Improve this question

3 Answers 3

Reset to default
1

I would recommend running the Craft queue in a background process as detailed here. This way it won't bog down your control panel and can just chug away in the background.

Improve this answer
3
  • Thanks for the recommendation! I'll have a look at doing that, do you think that could be what the issue is?
    – Moucky
    Commented Nov 19, 2019 at 18:53
  • It's hard to say but 15,000 entries is a lot to import, so it will likely take a while (15 minutes-ish is my guess) depending on how many fields in needs to compare.
    – dmatthams
    Commented Nov 20, 2019 at 14:50
  • The actual amount of entries I was trying to import there was only 58, the amount current entries on the site is around 15,000. I think there will be a total of about 30,000 when all the importing is complete but I do them per catalogue. Some have less than 100, others may have about 300.
    – Moucky
    Commented Nov 20, 2019 at 19:07
1

If one single update query (on one record) is taking over 2 minutes, I would guess that the bottle neck is the database. You could check if this is an incident or if these update queries are structurally slow.

Improve this answer
3
  • Thanks for replying. What you're saying is my concern, there is some issue with the database having lots of entries(even though I didn't think it was enough to cause trouble). This log is from the third Feed Me step, before any importing even happens, so it's querying the database for the entries before I map the fields. I'm not sure how to check any further what's causing the problem.
    – Moucky
    Commented Nov 20, 2019 at 19:13
  • What happens if you execute the INSERT-query manually on the database through the command line or a database tool? The 'ON DUPLICATE'-part is important as well. It checks if the key already exists and performs an update instead of an insert if the key already exists. That does not make it faster but is is probably the fastest solution. You could try adding an index on that table, in case there is no index yet.
    – Niek
    Commented Nov 21, 2019 at 20:42
  • Is it the SELECT query or the INSERT one that's taking a long time? My understanding was that the times shown are "Since Previous" meaning the SELECT query would be the slow one. Anyway I just tried executing the SELECT and INSERT queries manually(using PhpMyAdmin) and the were both very quick. SELECT was 0.0004 seconds and the INSERT was even less.
    – Moucky
    Commented Nov 21, 2019 at 22:02
1

Seems the answer that worked for me was to not use Structure section types for large amounts of entries. I ended up recreating the Structure section as a Channel and re-imported the entries. I feel as though it should be mentioned in the documentation that you should not use Structures if you plan on having many children entries, even at just two levels. Perhaps others would assume this is the case but I had no way of knowing.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.