2

Can you let me know what the process would be? I heard there was a no track tag we can put in the site header so the sites won't show up in search engines.

6

If you're used to another CMS such as WordPress, you're probably used to using a Discourage search engines from indexing this site option. As Robin's answer mentions, the training wheels are off.

However, that doesn't mean you can't replicate WP's functionality.

When you tick that box, there's a few things that happen behind the scenes:

  • If robots.txt file is requested, WP auto generates a version that tells google to skip indexing the site.
  • A <meta name="robots" content="noindex,follow"> is added to each page.
  • XML-RPC pinging is disabled (so WP won't "ping" services like ping-o-matic when you update pages on the site).

Craft doesn't automatically ping any services so you don't have to worry about the last option but the first 2 are pretty straightforward to setup.

Global Lightswitch / templates

Create a Global called siteSettings and add a Lightswitch with a field handle of disableIndexing so you can reference throughout the site.

In the <head> section of your templates, it can be as simple as:

{% if siteSettings.disableIndexing %}
    <meta name="robots" content="noindex,follow">
{% endif %}

robots.txt

The second piece is some Craft magic. If you don't already have a robots.txt file, create one inside Craft's templates folder or remove it/move it if it exists in your public root folder.

Craft is smart enough to parse your robots.txt file as Twig code and even output the proper mime type. This allows you to dynamically change it as it is requested.

When robots.txt is requested, instead of serving the file directly as your web server would, Craft will instead take the request and because there is a file that matches robots.txt inside templates folder, it'll use it.

Put this in your robots.txt file:

User-agent: *
Disallow: {% if siteSettings.disableIndexing %}/{% endif %}

We're basically telling search engines not to index anything when the Lightswitch is ticked. That's a pretty basic robots.txt file but it gets the job done for this example. There's a ton more options.

Wrap up

That's really it. The only thing to remember is to UNTICK the box when your site goes live or when you want search engines to revisit.

Lots of people in WP land get burned because they forget to turn it back on so this isn't really a great thing to do in the first place but this answer goes into some basic Craft concepts.

If you want more control, instead of a Global lightswitch, you could setup one up on a per entry basis much like Yoast does. This is pretty common for landing pages used for PPC so they don't get indexed to begin with. Just remember not to include them in your sitemap as well. Also consider the SEOMatic plugin which can do all of this and more.

1

Crafts does absolutely nothing in your frontend by itself unless you use certain twig functions (that mostly render css/js files). So it's totally up to you what seo options you provide. It's note like other cms that render your headers or footers or create non editable links thus you have the freedom to define everything by yourself.

So if you don't want to appear in search engines just don't include seo optimisations. If your question is about how to do that you are in the wrong board here because that is off topic since it's not related to craft

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.