Converter · SOTA RFC 9309 · per-bot matching
Robots.txt Tester & Parser
Test robots.txt against RFC 9309. Simulate per-URL allow or deny decisions for Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, and any custom user-agent product token. Inspect the matched rule, sitemap declarations, and crawl-delay values.
How to Use Robots.txt Tester in 3 Steps
- Configure. Paste the robots.txt content you want to test.
- Process. Enter a URL path to test (for example, /admin/login). Choose a search crawler, user-triggered agent, training crawler, or custom user-agent product token.
- Export. See the decision (Allowed / Blocked) with the exact rule that matched and its line number. Also shown: all sitemap declarations and per-bot crawl-delay.
Why Robots.txt Tester on Pixlane
Robots.txt tells compliant crawlers which URL paths they may fetch. A mistaken Disallow: / can stop a crawler from reading the entire site, while blocking crawl does not by itself guarantee URL removal from an index. Pixlane implements RFC 9309 longest-match resolution and lets you test Google, Bing, OpenAI, Anthropic, Perplexity, and custom product tokens separately.
- RFC 9309 matching: Resolves the most specific matching allow or disallow rule for the selected product token and path.
- Purpose-aware presets: Distinguishes search crawlers such as OAI-SearchBot and Claude-SearchBot, user-triggered fetchers such as ChatGPT-User and Claude-User, training crawlers such as GPTBot and ClaudeBot, and Google-Extended, which is a robots.txt product token that does not affect Google Search ranking or inclusion.
- Sitemap + Crawl-Delay: Extracts Sitemap declarations and per-user-agent Crawl-delay values, while flagging non-standard directives such as Host and Noindex.
- Pattern tester: Supports wildcards (*), end anchors ($), and case-sensitive path matching so you can inspect the exact winning rule.
Frequently Asked Questions
What is RFC 9309?
RFC 9309 is the IETF standard for the Robots Exclusion Protocol. It defines how crawlers select user-agent groups, match paths, and resolve allow and disallow rules.
What is the difference between Googlebot and Google-Extended?
Googlebot crawls content for Google Search. Google-Extended is a separate robots.txt product token that controls certain Gemini and Vertex AI uses; it is not a separate HTTP user agent and does not affect Google Search inclusion or ranking.
Which OpenAI crawler should I test?
Use OAI-SearchBot for ChatGPT search discovery, ChatGPT-User for user-triggered page visits, and GPTBot for model-training controls. They serve different purposes and should be tested separately.
Does robots.txt prevent indexing?
No. Robots.txt controls crawling. A URL blocked from crawling may still be known from links; use a crawlable noindex directive when URL removal is the goal.
Is this tool free?
Yes. Robots.txt Tester on Pixlane is free with no signup required.