1

In Craft 3, I keep running into an issue when I try to move a large amount of files (around 200) by dragging and dropping from my “Holding” folder into a “Live” folder using Craft Assets section. Both folders are hosted on Amazon S3. Even moving three files seems to take a really long time (about a minute).

After checking the server logs it looks like I am filling my max apache connection due to the search-index-perform-action. The error suggested I increase my “MaxRequestWorkers” setting.

I move about 2,000 files every month and never had a problem on Craft 2. I don’t know if should change the “MaxRequestWorkers” setting or if there is something else I should consider doing. If I should change the setting, where is it, I can't seem to find it.

I use Transmit FTP to initially move the files into the Holding folder and then index the assets in that folder. Then in Craft CMS control panel, I drag the files from Holding folder to the main Live folder so that the asset index gets updated.


UPDATE

I looked in my phperror.log file and found these three types of errors occuring:

PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 1048576 bytes) in /storage/avxxxxx/www/public_html/vendor/craftcms/cms/src/helpers/DateTimeHelper.php on line 231

PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /storage/avxxxxx/www/public_html/vendor/craftcms/cms/src/elements/db/ElementQuery.php on line 2215

PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem/src/Util.php on line 322

PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php on line 294

PHP Fatal error:  Maximum execution time of 90 seconds exceeded in /storage/avxxxxx/www/public_html/vendor/league/flysystem/src/Util/ContentListingFormatter.php on line 52

It seems like when I move a file from Holding (950 files) to Live (20,000) files is where things get hung up.

  • Holding -> Live = 1 file takes 7 seconds to move
  • Live -> Holding = 1 file takes 20 seconds to move

Could it be an issue with the cache or search-index?

Here are my current settings if it helps:

memory_limit 256M

max_execution_time = 90

Max Requests = Per Child: 300 - Keep Alive: on - Max Per Connection: 150

Timeouts = Connection: 10 - Keep-Alive: 2
0

I would remove the overhead of the CMS all together by using an app like Filezilla, Cyberduck or similar app to move files. There are plenty of tutorials out there on how to connect to an S3 bucket using one of these apps (eg. https://stackoverflow.com/questions/45293159/how-to-setup-filezilla-pro-to-access-to-my-s3-bucket)

After you do this you will most likely need to update your asset indexes in the control panel:

Utilities → Asset Indexes (or Settings → Update Asset Indexes if you're on Craft 2)

| improve this answer | |
  • I am using Transmit to transfer the actual files and then updating the index of the files in the Holding folder linked in Craft. The problem is when I move the files from Holding to the permanent folder. I have the Holding folder so I can index only 2,000 at a time vs the permanent folder which grows to over 26,000 by years end. – Lori Jan 4 '19 at 22:32
  • I've been doing some research on Assets, and I thought this article (vaersaagod.no/en/…) by Andre Elvan may be helpful in understanding assets and image transforms in relationship to the server. It may or may not answer the question you posed here, but it's a good overall reference on the subject. – Vaughn D. Taylor Jan 8 '19 at 16:50
  • Thanks @vaughn-d-taylor . I read through the document you referenced and looked into the setting referenced but it seems like I'm ok there. My max_execution_timeout is 90 and my max_requests look high enough too. My memory_limit is 256 so that looks good. It takes about 10 seconds to move 1 file from Holding to Live if the asset folder located on my server. It's taking 20 seconds to move 1 file from Holding to Live if the asset folder is located on S3. Could it be that my search-index is so large that it's taking a while to update it with the new asset path? – Lori Jan 9 '19 at 16:13
  • I cannot say for sure if it's your search index causing the issue. However, you may try increasing your max_execution_time = 300, max_input_time = 600, and memory_limit = 4G (or more if you have it) just as an experiment. I've run into issues with a very large DB and CraftCMS. The only way I could deal with the issue was to crank up those settings. – Vaughn D. Taylor Jan 9 '19 at 18:29
  • My server company updated the items as @vaughn-d-taylor suggested. I tested it out with one file and the speed to transfer took 50% less time the first time but then the second time it took close to 2 minutes. My phperrors.log file didn't show any time out errors. But my queue.log file showed a lot of craft_queue and craft_templatecaches and craft_templatecachequeries. The craft_queue has delay as part of the query. – Lori Jan 9 '19 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.