The first step to diagnose would be to determine where the bottleneck is.
If you enable devMode, you'll see database profiling information in your browser's console, including the SQL executed for each query along with the total number of queries executed.
Those queries will be ordered by the slowest performing ones at the top. If there are multiple, slow performing queries that are taking up the majority of the time it takes to load the page, then there is likely something we could do in Craft to optimize performance in your scenario and you should contact us at firstname.lastname@example.org and send along a database dump so we can try to reproduce.
If the database queries look like they are not the issue, then the next likely culprit is disc access times. If these folders/files are in the cloud (S3, Rackspace, etc.), or you're using NFS for your file system, then those could be slowing down the request as well.