Robots.txt checker for search and AI crawlers

What your result means

Start with the first group. A blocked Googlebot means Google can’t read the page at all, for ordinary results or for its AI Overviews. A blocked OAI-SearchBot keeps the page out of ChatGPT’s search answers. Either way, a buyer asking for managed IT support in Bristol won’t be shown your page.

The second group fetches a page because a person asked a question. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. For those two, “Blocked” is a request they may not honour. Anthropic says its bots, Claude-User included, honour robots.txt.

Which crawlers should a service business let in?

Everything in the first group. Those are the crawlers that let Google, Bing, ChatGPT, Claude, Perplexity and Apple find your service and office pages in the first place.

Training crawlers are a business decision. OpenAI, Anthropic and Apple each use a separate token for training, so blocking GPTBot, ClaudeBot or Applebot-Extended doesn’t remove you from their search answers. Apple notes that Applebot-Extended doesn’t crawl pages itself; it only controls how Applebot’s pages are used.

Google-Extended sits in the first group for a reason. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”, but it does control whether Gemini may train on your pages and use them for grounding. Block it, and Gemini loses both.

Never use robots.txt to hide a customer portal. Anyone can read the file, and its rules are only requests, so listing /portal/ advertises that it exists. Put private areas behind a login.

How the checker works

1. It finds the group for each crawler. Crawler names match in any case, and several groups naming the same crawler are merged. A crawler without a group of its own uses the User-agent: * group, and if there isn’t one, nothing restricts it. Applebot uses Googlebot’s group when it has none, which Apple documents.
2. It picks the deciding line. Among the Allow and Disallow lines that match the start of the path, the longest wins, and Allow beats Disallow on a tie. * matches any run of characters and $ marks the end of the URL. Paths are case-sensitive, so /Services and /services differ, and anything after a ? counts too.
3. /robots.txt can always be fetched.

These rules follow RFC 9309 and Google’s robots.txt documentation.

Google reads only the first 500 KiB of a robots.txt file. The status code your server returns for the file matters as much as what it says, and a missing file is far safer than a broken one.

A worked example

The checker starts with this file, for a made-up IT support company, and the page /portal/help/reset-login. Take Googlebot.

1User-agent: *
2Disallow: /portal/
3Allow: /portal/help/
4
5User-agent: GPTBot
6User-agent: CCBot
7Disallow: /
8
9Sitemap: https://www.example.co.uk/sitemap.xml

No group names Googlebot, so it uses the * group. Two lines match the start of the path: line 2 (/portal/, 8 characters) and line 3 (/portal/help/, 13 characters). Line 3 is longer, so Googlebot may read the page. Change the path to /portal/invoices and only line 2 matches, so it is blocked. GPTBot and CCBot share lines 5 to 7, so they are blocked from the whole site.

Blocked doesn’t mean unindexed

Robots.txt stops crawling, not listing. Google can’t read a blocked page, but the URL can still appear in results if other sites link to it. To keep a page out of Google, let Googlebot crawl it and give it a noindex tag. A Noindex line in robots.txt is not supported.

Being crawlable is only the start. Your service and location pages then need to be indexed, to show up for what buyers search, and to be named when they ask AI. Our guide to SEO for service businesses covers what to do first, and getting named in ChatGPT covers the AI side. If your site runs on WordPress, our WordPress page shows where its robots.txt is edited.

FAQ