robots.txt Policy Lint — AI Crawler Index

Take a robots.txt you already have and say what it actually does. It lints the file against RFC 9309 and reports the errors that silently change meaning, answers whether a named crawler may fetch a named path and which rule decided it, audits which AI crawlers the file really stops (and which it only appears to), diffs two versions by EFFECT rather than by line, and merges a ready-made stance into an existing file without discarding the rules already there. It reads the file you paste; it fetches nothing. robots.txt is a request, not an enforcement mechanism, and the audit says so where a crawler is known to ignore it. Deterministic and read-only: there is no model behind it — every answer comes from a public dataset rebuilt every six hours from each operator's own published documentation and IP ranges, and the same skills are also available as MCP tools at https://www.pathwren.workers.dev/mcp/robots. No key, no signup, no quota. Independent and unaffiliated with any operator it documents.

Skills

• Lint a robots.txt
  Parse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, BOM. Each finding carries the line number and the fix.
  robots.txtlintrfc9309validation
• Would this crawler fetch this path?
  Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree.
  robots.txtcrawlerspathrules
• Which AI crawlers does this file actually stop?
  Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead.
  robots.txtai crawlersauditpolicy
• Diff two robots.txt by effect
  Compare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers 'did my edit do what I meant, and did it do anything else', including sitemap additions and whether the parse errors went up or down.
  robots.txtdiffreviewchange
• Add a ready-made stance to an existing file
  Merge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-tools, allow-all, maximum-ai-visibility) into a robots.txt you already have, without touching a single rule you wrote: a token you already name keeps your rules and the stance's version is reported instead of applied.
  robots.txtpolicygeneratormerge

How to call

https://www.pathwren.workers.dev/a2a/robots

Listed in

Directories this entry was found in.