2

Hello my website is not being indexed by Google. In Google Lighthouse it says:

Page is blocked from indexing

Blocking Directive Source X-Robots-Tag: none

However I cannot find anything that could be causing the blocking. There is nothing in the .htaccess file that would point to anything and I don't have a robots.txt file and in the head of the pages I have:

<meta name="robots" content="index,follow" />
<meta name="GOOGLEBOT" content="INDEX, FOLLOW" />

If you want to take a look the website is the-ropewalk.co.uk and is built using Craft CMS 3x. It's not a new site at all. It's about 4 years old and it was indexed at some point so something has changed. I did a full search on the site and indeed some of the templates used for the shop had robots set to no-follow but those were only used on the shop and not form the rest of the site or home page. I hyave changed those but still no joy.

Is there any settings that CraftCMS has that I might have accidentally set that might mean the site is not visible to Google?

What else should I check please?

Any pointers would be helpful.

Thank you.

Improve this question

2 Answers 2

Reset to default
3

Craft will set a X-Robots-Tag: none header for front end requests, if any of the following are true:

  1. The disallowRobots general config setting has been set to true in your config/general.php file

  2. The request URL includes a valid token (as per the tokenParam config setting (default is 'token').

  3. The request URL includes a valid preview param x-craft-preview or x-craft-live-preview (note: this only applies for Craft 4.12+ and 5.4+, which are upcoming releases at the time of writing this).

If neither of these are the case, you might have a plugin installed that is setting the X-Robots-Tag: none header. For example, I believe SEOMatic will set it if Craft's devMode config setting is set to true (which should not be the case for sites in production, for a plethora of reasons).

Improve this answer
4
  • 1
    Thanks very much for your reply. It helped very much. disallowRobots wasn't set to true and I didn't have any plugins that interfered with devMode. In general.php I had 'devMode' => false when maybe it should have been $isProd (not sure if that made a difference) but what I think was the problem was in .env (which I didn't realise) I had ENVIRONMENT=dev when it should be ENVIRONMENT=production. It is now being indexed. Your pointers set me on the right path.
    – cannon303
    Commented Aug 17 at 22:24
  • @cannon303 Ah, happy to hear that you sorted it out! Getting those environments mixed up is a common gotcha for sure.
    – Mats Mikkel Rummelhoff
    Commented Aug 17 at 23:39
  • I like to ->disallowRobots(App::env('CRAFT_DISALLOW_ROBOTS') ?? false) on config/general.php then add CRAFT_DISALLOW_ROBOTS=true to .env
    – Leo Leoncio
    Commented Aug 21 at 15:00
  • @LeoLeoncio Env vars prefixed with CRAFT_ will actually override whatever is in your general.php file, so in practice the above is completely redundant – i.e. if you removed disallowRobots from your config file altogether (the default is false) and had CRAFT_DISALLOW_ROBOTS=true in your .env file in applicable environments, you'd have the exact same result. See craftcms.com/docs/5.x/configure.html#environment-overrides
    – Mats Mikkel Rummelhoff
    Commented Aug 21 at 21:49
1

Here's how I approach this (without seomatic)

.env

CRAFT_ENVIRONMENT=dev

CRAFT_DEV_MODE=true
CRAFT_DISALLOW_ROBOTS=true

config/custom.php

<?php

use craft\helpers\App;

$isProd = in_array(App::env('CRAFT_ENVIRONMENT'), ['prod', 'production', 'live']);

return [
    'isProd' => $isProd
];

_layout.twig

{% set isprod = craft.app.config.custom.isProd ?? false %}

inside <head></head>

{{ not isprod ? tag('meta', {
    name: 'robots',
    content: 'noindex'
}) }}
{{ not isprod ? tag('meta', {
    name: 'googlebot',
    content: 'noindex'
}) }}

config/routes.php

<?php

return [
    'robots.txt' => ['template' => 'routes/robots.txt']
];

templates/routes/robots.txt.twig

{% set isprod = craft.app.config.custom.isProd ?? false %}
{% header "Content-Type: text/plain; charset=utf-8" %}
{% if isprod %}
# live
User-agent: *
Disallow: /admin/
Disallow: /cpresources/
Disallow: /cache/
sitemap: https://my-site.com/sitemap.xml
{% else %}
# lower env
User-agent: *
Disallow: /
{% endif %}
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.