Test robots.txt against RFC 9309 (2022). Per-URL allow/deny simulation for Googlebot, Bingbot, GPTBot, ClaudeBot, YandexBot, and custom user agents. Detects sitemap, crawl-delay, and comment syntax.
Result
Parsed robots.txt
How to Use Robots.txt Tester in 3 Steps
Configure. Paste robots.txt content or enter a domain and let the tool fetch /robots.txt for you.
Process. Enter a URL path to test (e.g., /admin/login). Pick a user agent: Googlebot, Bingbot, GPTBot, ClaudeBot, YandexBot, or custom.
Export. See the decision (Allowed / Blocked) with the exact rule that matched and its line number. Also shown: all sitemap declarations and per-bot crawl-delay.
Why Robots.txt Tester on Pixlane
Robots.txt controls which URLs crawlers can access. A misconfigured file can deindex your whole site (Disallow: /) or leak URLs you didn't mean to expose. Pixlane implements RFC 9309 — the 2022 official robots.txt standard — with longest-match rule resolution and dedicated bot simulation for Google, Bing, OpenAI, Anthropic, Yandex, and custom bots.
RFC 9309 Compliant — The 2022 official IETF standard for robots.txt. Uses longest-match rule (the most specific pattern wins), not first-match. Matches Google/Bing behavior.
AI-Bot Aware — Pre-configured user agents for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot (Common Crawl), and PerplexityBot — the 2024 AI training agents you likely want to control.
Sitemap + Crawl-Delay — Extracts all Sitemap: declarations and per-user-agent Crawl-delay values. Flags non-standard directives (Host:, Noindex:) with explanations.
Pattern Tester — Supports the full Google-extended syntax: wildcards (*), end-anchor ($), and case-sensitive path matching. See exactly which rule matches your URL.
Frequently Asked Questions
What is RFC 9309?
The official robots.txt standard published by the IETF in September 2022. It formalized 28 years of conventions, including longest-match rule resolution. Before RFC 9309, behavior varied by crawler; now there's a single spec.
What's the difference between Googlebot and Google-Extended?
Googlebot indexes for search. Google-Extended (introduced 2023) controls whether your content is used to train Google's AI models (Bard, Gemini). Blocking Googlebot alone doesn't block AI training — you need Google-Extended too.
Does robots.txt prevent indexing?
No — it only controls crawling. To prevent indexing, use `<meta name="robots" content="noindex">` or X-Robots-Tag HTTP header. Pages blocked in robots.txt can still appear in search if linked from elsewhere.
Is this tool free?
Yes. Robots.txt Tester on Pixlane is completely free with no signup required.