Robots.txt Blocking Tests

CheckView can respect robots.txt rules and noindex directives, controlled by the Respect Bot Restrictions setting. It is off by default, so a new organization tests pages regardless of robots.txt and noindex. When it is turned on and your site’s robots.txt blocks the user agent CheckView’s test browser sends (a standard browser user agent unless you set a custom one), or blocks all user agents, or the page has a noindex meta tag, the test fails before it begins.

How It Works

When Respect Bot Restrictions is on, CheckView checks the site’s robots.txt file and the target page’s meta tags before running a test. If the page is disallowed or marked as noindex, the test fails with one of the following error codes:

• user-agent-disallowed: The site’s robots.txt file blocks the test browser’s user agent from accessing the page.
• noindex: The page has a noindex meta tag or HTTP header, indicating it should not be indexed or tested.

The Respect Bot Restrictions Setting

You can control this behavior in Organization Settings > Platform Configuration > Respect Bot Restrictions:

• Yes: CheckView respects robots.txt and noindex directives, and a test fails if the page is blocked.
• No (default): CheckView ignores robots.txt and noindex directives and tests the page anyway.

When to Disable Bot Restrictions

Consider disabling bot restrictions if:

• Your robots.txt broadly blocks crawlers but you still want to test your pages.
• Pages have noindex tags for SEO reasons but should still be tested for functionality.
• You are testing staging or development environments that block all bots by default.

Fixing robots.txt Issues

If you want to keep bot restrictions enabled but allow CheckView access, you can add a specific rule to your robots.txt:

User-agent: *
Allow: /

Or, to allow all bots access to specific paths only, add allow rules for the URLs you want to test.