Robots.txt Tester
Paste your robots.txt and test URLs against it.
100% in your browser — nothing uploadedTest which URLs your robots.txt allows or blocks
About this robots.txt tester
Paste your robots.txt file and test whether a specific URL path is allowed or blocked for a given user-agent. The tool parses the file following the same rules search engines use, finds the matching user-agent group, and applies the most specific matching rule. The result shows a clear allowed/blocked verdict plus the exact line that matched.
Robots.txt is the first thing crawlers check before requesting any page. A single misplaced rule can block your entire site from being indexed — or accidentally expose pages you meant to hide. Testing changes before deploying them prevents costly indexing mistakes that can take weeks to recover from.
The parser handles all standard robots.txt features: multiple user-agent groups, Allow and Disallow directives, wildcard (*) patterns in paths, end-of-URL anchors ($), and Sitemap directives. When two rules of equal specificity conflict, Allow wins — matching Google's implementation. User-agent matching is case-insensitive.
Built-in user-agent presets include Googlebot, Bingbot, and GPTBot, plus a wildcard (*) option and a custom field for any other crawler. The tool also extracts and displays any Sitemap directives found in the file. Everything runs in your browser — your robots.txt is never sent to a server.
FAQ
How does rule priority work?
The most specific (longest path) matching rule wins. If two rules have the same length, Allow takes precedence over Disallow — matching Google's documented behavior.
Does this support wildcards?
Yes. The * wildcard matches any sequence of characters, and $ anchors the match to the end of the URL. For example, Disallow: /*.pdf$ blocks all PDF files.
Is my robots.txt sent anywhere?
No. All parsing and testing runs locally in your browser. Nothing is uploaded.