Back to code-only MCP servers

crawler

active

polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.

Code-only MCP server: no hosted endpoint, so you install and run it yourself. It is not probed, graded or counted in our reachability statistics.

Get it

Repository
github.com/mark1russell7/crawler
Package
npm: polite-crawler
Run (stdio)
npx -y polite-crawler
The usual command for this package registry; check the repository README for required settings.
License
MIT
Last update
2026-11-15
GitHub stars
★ 0
Homepage
mark1russell7.github.io/crawler/

Listed in

Run this server as a hosted endpoint? Submit the endpoint and it moves to the remote directory, where it is probed and graded.