Robots.txt Tester
Check whether search engines and AI crawlers can access any URL on your site, test your robots.txt against real Google matching rules in seconds. Free.
- ✓ Real Google-spec matching
- ✓ Test any crawler (incl. AI bots)
- ✓ Fetch or paste robots.txt
- ✓ Instant allowed/blocked result
One wrong line in your robots.txt can accidentally block Google from your most important pages, and you might not notice for months. This free tester lets you fetch your live robots.txt (or paste one), then check whether any URL is allowed or blocked for a specific crawler, using the same wildcard and most-specific-rule logic Google actually applies.
Catch costly mistakes
Find out instantly if you’re blocking pages you meant to keep crawlable, before it hurts your rankings.
Test any crawler
Check Googlebot, Bingbot and AI crawlers like GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
Accurate matching
Supports wildcards (*), end-anchors ($) and Google’s longest-match-wins rule.
Test Your Robots.txt
Fetch your robots.txt (or paste it), then test any URL and crawler.
Matching follows Google’s rules (wildcards *, end-anchor $, most-specific rule wins). Free tool.
How It Works
Add your robots.txt
Enter your website and click “Fetch robots.txt”, or paste your rules directly.
Choose a URL & crawler
Enter the URL or path to test and pick the crawler (e.g. Googlebot or GPTBot).
See the result
Instantly see whether it’s allowed or blocked, and exactly which rule decided it.
What Is a Robots.txt File?
robots.txt is a plain-text file at the root of your site (e.g. example.com/robots.txt) that tells crawlers which parts of your site they may or may not access. It uses User-agent lines to target specific crawlers and Allow / Disallow rules to control access. Getting it wrong can hide your site from search engines entirely.
How Robots.txt Matching Works
For a given crawler, Google picks the most specific matching User-agent group, then applies the most specific (longest) matching rule, and if an Allow and Disallow rule are equally specific, Allow wins. Rules support the * wildcard (any characters) and $ (end of URL). This tester replicates that logic so your results match reality.
Robots.txt & AI Crawlers
As AI search grows, you may want to control access for AI crawlers too, GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended. Test how your rules apply to each, so you deliberately choose what AI engines can and can’t use. If you want to appear in AI answers, don’t block them, see our GEO service.
Explore More Free Tools & Services
Frequently Asked Questions
Is this robots.txt tester free?
Yes, testing is free and unlimited, and the matching runs in your browser. Fetching a live robots.txt has a generous daily fair-use limit.
What does “blocked” mean?
It means the crawler you selected is not allowed to crawl that URL under your current robots.txt rules. Blocked pages generally won’t be crawled (though they can still be indexed if linked elsewhere).
Can I test AI crawlers like GPTBot?
Yes, the crawler dropdown includes GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more, so you can see exactly how your rules apply to AI bots.
Does robots.txt stop a page being indexed?
Not by itself. Robots.txt controls crawling, not indexing, a blocked page can still appear in results if other sites link to it. To keep a page out of the index, use a noindex meta tag instead.
Where should my robots.txt live?
At the root of your domain: https://yourdomain.com/robots.txt. It only applies to that exact host and protocol.
Why Do It Manually?
A clean robots.txt is just one piece of technical SEO. Let our team audit and fix crawling, indexing and everything else, so search engines see your best pages.
Browse All Free Tools →Check My RankingRobots.txt Tester
The robots file tells crawlers where they may go. It is a few lines of text and it is capable of removing a whole site from search with one of them.
The classic failure is a staging site rule that goes live with the site. Everything looks fine, nothing gets indexed, and the cause sits in a file nobody has opened since launch.
It is worth being clear about what it does not do. Blocking a page stops it being crawled, not indexed. A blocked page can still appear in results, with no description, which is the worst of both.
How to use it
- Load your robots file and read it as it stands.
- Test the URLs that matter. Home, services, location pages.
- Look for a blanket disallow rule left over from development.
- Use noindex, not robots, for pages you want kept out of results.
Why it matters
This is a two minute check that occasionally explains months of confusion. It is worth doing after any site move or redesign, without waiting for a reason.
Questions people ask
Can robots.txt remove a page from Google?
Not reliably. It blocks crawling. A blocked page can still be listed without a description. Use noindex to keep something out.
What is the most common mistake?
A disallow all rule from a staging site shipped to production. It hides everything.
Do I need one at all?
Not necessarily. An absent file is safer than a wrong one.