A crawler block prevents reading noindex

What to change

A crawler needs to fetch a page to read its page-level instructions. Ask the supplier to explain whether the current robots rule prevents that fetch, then choose a coherent approach for the intended public resource. Keep private material behind genuine access controls. For a public page intended for search exclusion, the evidence should show that the relevant indexing instruction is reachable.

A worked example

Illustrative example

Illustrative test: Use a public obsolete help page, not a confidential file, for the diagnostic.

How to check the result

The allowed fetch can see the intended noindex and remaining restrictions are documented.

A mistake to avoid

Index controls are not a substitute for authentication of private information.

Your action checklist

Use these checks to prepare a discussion with your team or supplier. Tick a check when you have recorded the decision or evidence, rather than when a feature has simply been promised.

Decisions to recordRecord the page-level indexing instruction currently served to an ordinary request.Compare that address against the matching robots.txt path rule.Test whether the selected crawler can fetch the instruction after the change.

Evidence, open questions and decisionsNext action and responsible role

Entries stay in this page and are not submitted to Kay & Co. Download your notes before leaving; the page does not save them.