robots-txt-parser.apimesh.xyz

● down utility

Fetch, parse and analyze the robots.txt of a given website

How to call

cURL · x402-fetch / x402-axios will auto-handle 402

curl -X POST https://robots-txt-parser.apimesh.xyz

/.well-known/x402.json