3

Can you let me know what the process would be? I heard there was a no track tag we can put in the site header so the sites won't show up in search engines.

Improve this question

4 Answers 4

Reset to default
6

If you're used to another CMS such as WordPress, you're probably used to using a Discourage search engines from indexing this site option. As Robin's answer mentions, the training wheels are off.

However, that doesn't mean you can't replicate WP's functionality.

When you tick that box, there's a few things that happen behind the scenes:

  • If robots.txt file is requested, WP auto generates a version that tells google to skip indexing the site.
  • A <meta name="robots" content="noindex,follow"> is added to each page.
  • XML-RPC pinging is disabled (so WP won't "ping" services like ping-o-matic when you update pages on the site).

Craft doesn't automatically ping any services so you don't have to worry about the last option but the first 2 are pretty straightforward to setup.

Global Lightswitch / templates

Create a Global called siteSettings and add a Lightswitch with a field handle of disableIndexing so you can reference throughout the site.

In the <head> section of your templates, it can be as simple as:

{% if siteSettings.disableIndexing %}
    <meta name="robots" content="noindex,follow">
{% endif %}

robots.txt

The second piece is some Craft magic. If you don't already have a robots.txt file, create one inside Craft's templates folder or remove it/move it if it exists in your public root folder.

Craft is smart enough to parse your robots.txt file as Twig code and even output the proper mime type. This allows you to dynamically change it as it is requested.

When robots.txt is requested, instead of serving the file directly as your web server would, Craft will instead take the request and because there is a file that matches robots.txt inside templates folder, it'll use it.

Put this in your robots.txt file:

User-agent: *
Disallow: {% if siteSettings.disableIndexing %}/{% endif %}

We're basically telling search engines not to index anything when the Lightswitch is ticked. That's a pretty basic robots.txt file but it gets the job done for this example. There's a ton more options.

Wrap up

That's really it. The only thing to remember is to UNTICK the box when your site goes live or when you want search engines to revisit.

Lots of people in WP land get burned because they forget to turn it back on so this isn't really a great thing to do in the first place but this answer goes into some basic Craft concepts.

If you want more control, instead of a Global lightswitch, you could setup one up on a per entry basis much like Yoast does. This is pretty common for landing pages used for PPC so they don't get indexed to begin with. Just remember not to include them in your sitemap as well. Also consider the SEOMatic plugin which can do all of this and more.

Improve this answer
3

Craft 3.5.10 added a new disallowRobots config setting. If set to true, Craft will add a X-Robots-Tag: none HTTP header to frontend requests, indicating to search engines that the page should not be indexed or crawled.

Improve this answer
3
  • Beware, on a recent build, I found this tag/header to be enabled by default, and it took some time to track down why Google was not indexing the site.
    – 4midori
    Mar 17, 2022 at 22:52
  • The default value for the disallowRobots setting is (and has always been) false. It is enabled (i.e. set to true) in the official Composer starter project, though – unless the current environment does not have an environment variable ENVIRONMENT set to 'production'. Probably that is what you ran into?
    – Mats Mikkel Rummelhoff
    Mar 18, 2022 at 7:18
  • Not sure. On this build, I don't think I made any changes in general.php or .env. I used an unfamiliar web host, so this setting may have been incorrect from the start.
    – 4midori
    Mar 19, 2022 at 18:28
1

Crafts does absolutely nothing in your frontend by itself unless you use certain twig functions (that mostly render css/js files). So it's totally up to you what seo options you provide. It's note like other cms that render your headers or footers or create non editable links thus you have the freedom to define everything by yourself.

So if you don't want to appear in search engines just don't include seo optimisations. If your question is about how to do that you are in the wrong board here because that is off topic since it's not related to craft

Improve this answer
1

You could also use this code in robots.txt file placed inside Craft's template folder.

{# Disallows #}
{% if craft.app.config.env != 'production' %}

{# Disallow access to everything when NOT in production #}
User-agent: *
Disallow: /

{% else %}

{# Disallow access to cpresources/ when live #}
User-agent: *
Disallow: /cpresources/

{% endif %}
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.