Paste a robots.txt, a URL path, and a user-agent. The tester returns an Allowed or Blocked answer with the matching rule shown. It follows the RFC 9309 standard (longest match wins, Allow beats Disallow on tie, * and $ wildcards supported).
Paste the contents of your robots.txt file.
Enter the URL path and pick a user-agent.
The tester implements the rules from RFC 9309, which is the formal robots.txt standard. Key rules:
* group.* matches any sequence of characters (including none). $ anchors the rule to the end of the path.Per RFC 9309, the user-agent in the request is compared against the user-agent tokens in each group. The longest matching token wins. The comparison is case-insensitive and uses substring matching against the agent product token. If no specific group matches, the User-agent: * group applies.
These are robots.txt directives, but they're not part of the Allow/Disallow decision. The tester parses your robots.txt but only uses Allow / Disallow rules for the verdict. Sitemap: lines are not matched against any UA group. Crawl-delay: is recognized by some crawlers but not all. Host: is recognized by a small number of crawlers and is not part of RFC 9309.
It means a more specific rule overrides a more general one. Example: Disallow: /admin/ blocks /admin/users, but if you also have Allow: /admin/public/, then /admin/public/list is allowed because the Allow rule has a longer matching path (14 chars vs 7 chars).
No. The tester only operates on text you paste. It never makes network requests to your site. If you want to test your live robots.txt, copy it from https://yoursite.com/robots.txt and paste it here.
Robots.txt blocks crawling, not indexing. If a search engine sees your URL linked from other sites, it can include the URL in its results (without content) even if it never crawled the page. To prevent indexing, use a noindex meta tag or HTTP header, and don't block the URL in robots.txt (otherwise the search engine can't read the noindex).
No. Everything runs in your browser. The robots.txt content and the URL you test never leave your device.