How to Build, Validate, and Test robots.txt for Search and AI Crawlers

A robots.txt file is a public request placed at the site's top-level `/robots.txt` path. It tells compliant crawlers which URL paths they may fetch, but it is not an authorization system and should never contain a directory merely because its name is secret. RFC 9309 standardizes the Robots Exclusion Protocol, including user-agent groups, path matching, rule precedence, encoding, and retrieval behavior. Authentication remains the control for material that must stay private.

Search crawling, AI training, and user-requested retrieval are separate product decisions. OpenAI, for example, documents OAI-SearchBot for search, GPTBot for potential training, and ChatGPT-User for user-initiated actions. Other vendors publish their own agents and policies. Grouping every name under an undefined idea of an “AI bot” can accidentally block discovery you wanted or permit reuse you meant to decline, so record the purpose and source URL for each agent you add.

Inventory crawlable URL classes before writing directives. Create the broadest user-agent group that shares a policy, add narrower groups only when their treatment differs, and express paths from the URL root with the same capitalization used by the site. Test both a blocked path and a deliberately allowed exception. RFC matching uses the most specific rule; when Allow and Disallow have equally specific matches, Allow wins. Add absolute Sitemap locations separately, then fetch the deployed file as an anonymous visitor.

RFC 9309 governs general matching semantics. Google's crawler and robots documentation governs Google-specific limits and unsupported directives, while each AI vendor's current crawler page is the authority for that vendor's token and declared purpose.

Allow ordinary crawling but block `/private/`, then reopen `/private/public-guide` with a longer Allow rule. Add OAI-SearchBot to the allowed search policy and put GPTBot in a separate training group with `Disallow: /`. Test `/private/report`, `/private/public-guide`, `/Private/report`, `/assets/site.css`, and the homepage. The capitalized path is a distinct case, and blocking the training agent must not be described as removing an already indexed search result.

Decision order: select every group whose user-agent product token matches, merge matching groups, find the longest matching path rule, and prefer Allow on an exact-length tie. If no group matches, a wildcard group may apply.

Google ignores `crawl-delay`, and `noindex` is not a supported robots.txt directive. A disallowed URL can still appear in search without a snippet if Google discovers its address elsewhere; use an index-control method on a crawlable response or removal workflow when disappearance is the goal. Google limits robots processing to 500 KiB. Blocking CSS or JavaScript can prevent a crawler from rendering a page accurately, while listing sensitive paths advertises their names to everyone.

Test representative public, private, asset, and query-string paths for the wildcard group and every crawler with a specific policy. Watch for one recurring error: using one blanket AI switch even though search, training, and user-requested fetchers publish different tokens and may follow different rules.

Build user-agent groups from the reviewed catalog or enter a custom verified token, add Allow and Disallow paths, include sitemap URLs, and run the path tester against representative URLs. Download the text only after resolving whole-site, asset, and unsupported-directive warnings.

Input: crawler tokens, access rules, sitemap URLs, and test paths. Output: a UTF-8 robots.txt draft, diagnostics, and an explanation of the matching decision. The tool cannot deploy the file, authenticate a crawler, or remove a URL from an index.

Generation, imported-file parsing, validation, and path testing run locally without requesting the website. The builder creates RFC-style groups, preserves protected paths in explicit crawler groups, validates the editable source, and applies longest-match testing to a selected URL path.