Back to code-only MCP servers

TheCrawler

updated this year

Open-source web scraper + LLM-powered structured extraction. PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, 16 analytics-tracker detection. Structured errors with retryable flags. Adaptive Cheerio->Playwright. CLI, npm, REST API, and MCP server. AGPL-3.0.

Code-only MCP server: no hosted endpoint, so you install and run it yourself. It is not probed, graded or counted in our reachability statistics.

Get it

Repository
github.com/manchittlab/TheCrawler
Package
npm: thecrawler
Run (stdio)
npx -y thecrawler
The usual command for this package registry; check the repository README for required settings.
License
AGPL-3.0
Last update
2026-07-02
GitHub stars
★ 2
Homepage
miaibot.ai/tools/thecrawler

Listed in

Run this server as a hosted endpoint? Submit the endpoint and it moves to the remote directory, where it is probed and graded.