1

So I have some gated content entries and need to prevent these from being indexed by search engines etc. Currently using draft pages but the tokens expire which then causes a 404 if someone is to land on that entry.

Anyway to prevent these tokens from expiring? Or does anyone have any suggestions on a field to prevent specific entries from being indexed by search engines. Or any options for saving as hidden entry, would be nice if this was built into craft as it seems like a pretty common feature.

2

I ended up adding a lightswitch for do not index. I am checking if this value exists in _layout.html if it does I am adding

{% if entry.noIndex %}<meta name="robots" content="noindex,nofollow">{% endif %}

I also updated my search results, sitemap entry template to check for the same thing and not include it in the sitemap file or the search results. As well as the dynamic sitemap.xml file. I believe this should be good.

Also I extended the token expiration but I think with the above approach, we will just publish like normal and remove the token expiration.

1

Or does anyone have any suggestions on a field to prevent specific entries from being indexed by search engines.

You could create a sitemap file that excludes those entries (based on some field in the entry), and include that sitemap in your robots.txt. That should discourage well-behaved search engines.

Also, you could requireLogin on those pages, which would make them inaccessible to anyone not logged in (which would include bots).

  • That is what I was thinking, I thought of just adding a checkbox at the top of entries that just says don't index. Then if that value is true, add the noindex, nofollow meta tag to _layout.html. Then on my current sitemap just exclude drafts. This would require extending the token duration. – Bowenac Jul 13 '15 at 19:16
  • Can't require login, most of this content users are being sent to based on if they filled out a form. Really wish there was a built in way to just create hidden entries haha. – Bowenac Jul 13 '15 at 19:34
0

Anyway to prevent these tokens from expiring?

You could set the defaultTokenDuration to some really high value. Just keep in mind this affects all tokens on the site including the ones used for email account verification and password resetting.

Or does anyone have any suggestions on a field to prevent specific entries from being indexed by search engines.

If you know the URLs, you could list them in nofollow.

  • If you know which path they'll be published under you could disallow /sectionName/* in robots.txt – megatrond Jul 13 '15 at 18:10
  • I still need the links to work, so I could set the tokenduration really high, then I guess if is draft nofollow. Are drafts also indexed? Or is that the purpose of the token so that the url is always different? – Bowenac Jul 13 '15 at 19:02
  • Also a specific path wouldnt work as some in the same path need to be indexed. I think I would need to check if it's a draft, add noidex meta tag. Or would I run into issues going that route for drafts that end up published? – Bowenac Jul 13 '15 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.