Robots.txt checker
One line in robots.txt can switch your whole site off in Google — and nothing on the site looks broken. Enter your domain and we fetch the file, parse every directive, and flag what a full audit flags: a stray Disallow: /, blocked CSS/JS, a missing sitemap line, blocked AI crawlers.
The file, the problems, the exact lines
Six problems, the same ones the £7.99 audit flags
This checker runs the robots.txt portion of our real audit engine — the rules behind the Crawlability and AI-Readiness findings in every report we sell.
1 · A stray Disallow: /
The one line that removes a site from Google entirely. Under User-agent: * it tells every crawler: fetch nothing. Usually a staging leftover or an SEO-plugin launch switch that never got turned off. Flagged as an error, line highlighted.
2 · CSS and JavaScript blocked
Google renders your pages before ranking them. If the stylesheet and script files behind your design are disallowed, the rendered page looks broken to Googlebot — and rankings follow the render. Flagged as a warning with every offending line.
3 · Blocked AI crawlers
GPTBot, ClaudeBot, PerplexityBot and friends told to stay away. As customers ask AI assistants for recommendations, a blocked crawler means your site can't be the answer. Flagged as a warning — a deliberate content decision is fine, an accidental one isn't.
4 · No Sitemap: line
robots.txt is where many crawlers look to discover your sitemap. Without the line, discovery of your newest pages is left to links alone. Flagged as a warning, with the exact line to add.
5 · Unrecognized user-agents
A typo'd User-agent: name silently disables the entire group beneath it — every real crawler keeps crawling as if the rules weren't there. Unknown names are flagged so you can check the spelling against the bot's published name.
6 · Crawl-delay misunderstandings
Googlebot ignores Crawl-delay completely — rate control for Google lives in Search Console. High values usually arrive from staging configs and only slow the crawlers that do honor them. Flagged as an informational note.
User-agent: *, Allow: /, and Sitemap: https://yourdomain.com/sitemap.xml.
How to read a robots.txt in 30 seconds
The file is just groups. Each User-agent: line starts a group; the Disallow: and Allow: lines under it are the rules for that bot. That's the whole grammar.
- Line 1 starts the rules-for-everyone group.
*means every crawler that doesn't match a more specific group below. - Lines 2–4 narrow the crawl, they don't kill it. Admin paths and internal search results are the two Disallows almost every site should have. Blocking resources the admin page loads would be a mistake — blocking the page itself is fine.
- Lines 6–7 are a per-bot group. GPTBot may crawl everything except the members area — a targeted AI policy, not a blanket ban.
- Line 9 declares the sitemap. An absolute URL, at the bottom of the file, so crawlers that only read robots.txt still find every page.
Robots.txt is one of 260+ checks
The checker above is free and instant. The full £7.99 audit runs all of it: 260+ checks across 21 categories on up to 25 pages, an AI-written plain-English report, delivered by email in ~30 minutes. What it costs · read a real sample report.
The full audit
Everything on this page plus the other 250+ checks, in one ordered fix-list.
- robots.txt presence, directives and what they actually block
- Sitemap exists, is valid, and is referenced from robots.txt
- AI-crawler access across GPTBot, ClaudeBot, PerplexityBot & friends
- Meta robots noindex tags and X-Robots-Tag headers, page by page
- llms.txt, structured data, and the other AI-readiness signals
- Everything else: SEO basics, accessibility, performance, security
— Not ours to claim
What no external tool can do:
- See inside Search Console — crawl stats and indexing requests live behind your Google login; the report names the exact screen to open
- Edit the file for you — the fix is yours to apply (or ours: we do fixes too); the report gives the exact lines
- Promise AI citations — nobody honestly can; what the report guarantees is that nothing on your site mechanically blocks the crawlers
Also honest: blocking some crawlers is a legitimate choice — the audit's job is to make the decision visible, not to reverse it.
Fair questions, straight answers
What should a robots.txt file contain?
A minimal healthy robots.txt is three lines: User-agent: *, Allow: / (or targeted Disallow rules for admin and internal paths), and Sitemap: https://yourdomain.com/sitemap.xml. The first two say who the rules apply to and what they may crawl; the last declares your sitemap where many crawlers look for it. Everything else — specific bot groups, Crawl-delay — is optional.
Is it bad to have no robots.txt at all?
It is not fatal — crawlers fall back to crawling everything, which is usually what you want — but it costs you control. Without robots.txt you cannot steer crawlers away from admin or search pages, you cannot declare your sitemap where many crawlers look for it, and the full audit counts the missing file as an error because crawl behaviour is uncontrolled. A three-line file fixes it.
Why is a lone "Disallow: /" so serious?
Under "User-agent: *", Disallow: / means no crawler may fetch anything on the site — Google drops your pages from results rather than ranking them badly. It usually arrives by accident: the WordPress "Discourage search engines" checkbox left on, a staging robots.txt deployed to production, or an SEO plugin's launch mode never switched off. The checker highlights the exact line so you can confirm whether it is deliberate.
Should I block AI crawlers like GPTBot in robots.txt?
That is a content-licensing decision, not a technical mistake — blocking them is valid if you do not want your content used for AI training or answers. But know the trade-off: as customers ask AI assistants for recommendations, a blocked crawler means your site cannot be cited. The checker flags blocked AI crawlers as a warning so the decision is visible, not accidental — which is exactly how the £7.99 audit treats it.
The other 259 checks are waiting.
Enter your URL and pay securely — your report PDF lands in your inbox in ~30 minutes,
with robots.txt, sitemap, AI-crawler access and 250+ other checks run and explained.