x402node.dev

https://api.x402node.dev

● healthy general

Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. -- reliability 91.0/100 per Paygent Discover

How to call

cURL · x402-fetch / x402-axios will auto-handle 402

curl -X POST https://api.x402node.dev

/.well-known/x402.json

https://api.x402node.dev/.well-known/x402