Robots.txt Tester & Validator
Verify whether Googlebot, Bingbot, or AI bots can crawl your URLs. Test rule precedence, syntax errors, and Allow vs Disallow specificity before deploying.
Crawler Simulator
Matched Disallow directive on line 3. Googlebot and matching crawlers will be blocked from accessing this URL.
Audit Robots.txt & Canonical Tags on Every Page
The OsaniRank Chrome Extension highlights crawl directives, indexability flags, and Moz DA right in your browser toolbar with zero friction.
How the Robots Exclusion Protocol Works (RFC 9309)
The Robots Exclusion Protocol (formalised by Google, Bing, and the IETF as RFC 9309) is the web standard that instructs automated web crawlers which URLs they may access on your domain. When a search engine like Googlebot attempts to crawl your site, the very first HTTP request it issues is to /robots.txt.
If this file permits crawling, the bot continues fetching HTML assets. If a URL matches an explicit Disallow: directive, Googlebot saves its crawl budget and refrains from requesting that file.
User-Agent Directives
Specifies which bot the following rules apply to. User-agent: * sets fallback rules for all crawlers, while specific names like Googlebot override general rules.
Allow & Disallow Paths
Paths match from the root URL. If multiple rules match the same test URL, Google selects the rule with the longest number of matching characters.
Sitemap Declaration
Points search spiders to your absolute XML sitemap URL. Unlike path rules, Sitemap: is universal and independent of User-Agent blocks.
Robots.txt Rule Precedence: Allow vs Disallow
Understanding rule precedence prevents fatal SEO errors where entire categories or product pages get accidentally blocked. Here is how Googlebot evaluates conflicting rules according to RFC 9309:
| Robots.txt Directives | Tested URL | Verdict | Reason |
|---|---|---|---|
| Disallow: /blog/ Allow: /blog/seo | /blog/seo | ALLOWED | Allow rule (/blog/seo, 9 chars) is longer and more specific than Disallow (/blog/, 6 chars). |
| Disallow: /admin/ Allow: /admin/ | /admin/dashboard | ALLOWED | Equal length match (7 chars each). RFC 9309 dictates Allow wins in an exact tie. |
| Disallow: /*?*sort= Allow: / | /shoes?color=blue&sort=asc | BLOCKED | Disallow pattern matches query parameter and is longer than root Allow directive. |
| Disallow: /private$ | /private/page | ALLOWED | Dollar anchor ($) matches only exact ending. /private/page does not end at /private. |
5 Critical Robots.txt Mistakes That Ruin SEO
Leaving Disallow: / in place when launching from staging or local development. This instantly wipes out your Google rankings and search visibility within 48 hours.
Blocking stylesheets or JS files prevents Google's rendering engine from confirming mobile responsiveness and above-the-fold content integrity.
Disallow prevents crawling, not indexing. If an internal or external link exists, Google indexes the bare URL. Always use a noindex meta tag for de-indexing.
Writing Disallow: admin instead of Disallow: /admin causes RFC 9309 compliant crawlers to treat the rule as invalid or inactive.
Related Technical SEO Tools
Sitemap Generator
Generate clean sitemaps.org 0.9 XML sitemaps with custom priority & lastmod.
URL Extractor
Extract, filter, and deduplicate all URLs from raw text, HTML, and markdown.
Google SERP Simulator
Preview Google desktop & mobile snippet layouts with pixel counters.
Frequently Asked Questions
Start dominating search rankings today.
Join over 50,000+ SEO professionals, content creators, and agencies using OsaniRank to accelerate their organic growth.