I dont understand why we need the
robots.txtfile in the Craft CMS package. I think it is confusing. (I had a problem with Google indexing one of my BETA hidden website because of that.)
Reason : When I initiate my domain name on the hosting it creates automatically a public folder inside the virtual host main folder.
So for example my hosting server make the domain name point to
/lamp0/web/vhosts/public/. So I just have to upload
public folders below in
Then I just have to rename
htdocs and no need of
robots.txt file because
craft folder is below domain name root.
# Don't allow web crawlers to index Craft User-agent: * Disallow: /craft/
OK I see now in the Craft install documentation (see below). I missed it sorry.
Perhaps would it be good to insert a
README.txt file in the package next to robots.txt file to explain that again :)