Agent Tools
Back to MCP servers

AI Crawler Index

https://www.pathwren.workers.dev/mcp?s=pathwren-edge&c=mcp-registry
● healthy

Identify AI crawlers by user-agent or IP, and build the robots.txt that matches your stance.

Transport
streamable-http
Auth
—
Cost
—
Protocol revision
2026-07-28
negotiated, not declared

Listed in

Directories this entry was found in.

mcp-registry

How to connect

MCP endpoint (streamable-http)
https://www.pathwren.workers.dev/mcp?s=pathwren-edge&c=mcp-registry
JSON-RPC initialize probe
curl -X POST https://www.pathwren.workers.dev/mcp?s=pathwren-edge&c=mcp-registry \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}'
Homepage
https://www.pathwren.workers.dev/mcp.html