I dont understand why we need the robots.txtfile in the Craft CMS package. I think it is confusing. (I had a problem with Google indexing one of my BETA hidden website because of that.)

Reason : When I initiate my domain name on the hosting it creates automatically a public folder inside the virtual host main folder.

So for example my hosting server make the domain name point to /lamp0/web/vhosts/public/. So I just have to upload craft and public folders below in /lamp0/web/vhosts/.
Then I just have to rename public in htdocs and no need of robots.txt file because craft folder is below domain name root.

enter image description here

# Don't allow web crawlers to index Craft
User-agent: *
Disallow: /craft/

OK I see now in the Craft install documentation (see below). I missed it sorry.
Perhaps would it be good to insert a README.txt file in the package next to robots.txt file to explain that again :) enter image description here

  • 2
    Most probably don't. However, not all hosting providers allow users to edit files above the webroot (or some users just prefer both /craft and public files in the same directory), in which case providing a default robots.txt file preconfigured to disallow /craft/ folder indexing is probably a good security measure (especially for those that don't know such a thing even exists). But feel free to add it as a suggestion to http://feedback.buildwithcraft.com/ – Douglas McDonald Aug 12 '15 at 17:00

I think it's there to protect those who can't have craft/ below web root. In your case it can't be found by search engines, so you can delete it.

| improve this answer | |

The specific directive to disallow the /craft/ directory isn't necessary if your craft folder is below the webroot.

However, it's good practice to have a robots.txt for each site, so I don't recommend deleting it altogether. You can change the directives to fit whatever robots criteria you need for your site. (For example, if you launch a beta site, you'd want to Disallow the whole thing to avoid duplicate content penalties.)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.