AI Crawler Index

Name any bot user-agent, verify a crawler IP, or generate robots.txt. Free, no key, no signup. Direct endpoint, no gateway: https://www.pathwren.workers.dev/c/smithery-direct/mcp (streamable HTTP, no key, no signup). Tools: classify_user_agent, lookup_crawler, list_crawlers, generate_robots_txt, is_verified_crawler_ip, whats_changed, example. Call the `example` tool with no arguments for a worked answer against real data before you write any code. What it does and how, in one page: https://www.pathwren.workers.dev/c/smithery-direct/mcp.html

Listed in

Directories this entry was found in.

How to connect

https://server.smithery.ai/pathwren/ai-crawler-index/mcp

curl -X POST https://server.smithery.ai/pathwren/ai-crawler-index/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}'