MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
cockroach-crawlergithub.com/ajnasnb/cockroach-crawlerupdated this year
Give AI agents superpowers on the web: crawl, render, map, extract, and verify evidence through one governed Node.js crawler.
npm MIT -
Fast, low-token web reader and focused crawler for AI assistants, served over MCP. Returns ranked passages instead of whole pages.
cargo Apache-2.0 ★ 7 -
Run the fastCRW web crawler natively on Windows: tray on/off switch, JS rendering via Chrome, MCP for AI agents, optional sharing over Tailscale.
MIT -
AI-powered web crawler — self-hosted Firecrawl alternative. Crawl, extract, render JS, search. CLI + MCP + REST API + Dashboard.
npm Apache-2.0 -
Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.
npm MIT ★ 266 -
Native-speed, browserless web crawler + MCP server for AI agents. A standalone Rust engine (turbo-dom + a V8 JS-render tier) — fetch + parse + extract + run page JS with no headless browser. This npm package is a thin launcher that spawns the native binar
npm MIT -
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
npm MIT -
AgentReady REST API and Docker deployment for AI agent readiness scanning. Scan websites, APIs, ecommerce stores, marketplaces, MCP servers and agent services for agentic-web compatibility, discoverability, OpenAPI, llms.txt, structured data, security and AI crawler readiness. Easy to self-host.
Apache-2.0 -
Safe, self-hosted web grounding for AI agents and crawlers — stealth Chrome over MCP
pypi MIT ★ 1 -
MCP server for technical & on-page SEO audits — crawlability, HTTPS, Core Web Vitals, sitemap, robots.txt, structured data, and AI-crawler access
npm MIT ★ 1 -
crwgithub.com/fastcrw/crwactive
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
cargo AGPL-3.0 ★ 1,260 -
AI-powered web crawler that captures designs and generates GSAP animation code (~$0.10/site). Built with Playwright, Claude AI, and MCP for seamless integration with Claude CLI.
★ 1 -
Playwright Web 控制台探索 + Android/iOS 接口证据提取(DEX、ELF .so、Mach-O)与受控验证 + MCP 网关。 Web UI crawler, APK/IPA native API evidence, mitmdump capture and a guarded MCP gateway for AI clients.
★ 28 -
AI-native web crawler and structured data extraction engine - one Rust binary with CLI, embedded dashboard, REST API and MCP server for AI agents.
MIT ★ 5 -
agent-seo-metaactive
Zero-dependency AIEO & GEO (AI Engine Optimization) auditor, AI crawler checker, and Schema.org JSON-LD generator for AI agents and modern web developers.
npm MIT -
Machine Realms (machinerealms.com) is a machine-native research commons and agent-network participant: a shared environment where humans, AI agents, services and crawlers meet in their native habitats — humans on the web surface, machines on a public HTTP+JSON index at /api/v1, a read-only remote MCP server at /mcp, an A2A 1.0 agent card with…
-
Open-source, index-free web search engine for AI agents and people. No crawler, no search API bill, no piggybacking on other engines. MCP server for Claude Code, Codex, Cursor and Gemini CLI. Self-hosted.
AGPL-3.0 ★ 2 -
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
cargo Apache-2.0 ★ 274 -
Free, open-source AI visibility and SEO audit. Crawl your whole site with 643 checks for technical SEO, AEO/GEO, 40 AI crawlers and agentic-web readiness (llms.txt, MCP, A2A), then track whether ChatGPT, Claude, Gemini and Perplexity name you. Fixes for your stack. CLI, dashboard and MCP server; runs locally.
AGPL-3.0 ★ 1 -
Web Crawler MCP
npm MIT -
@cd39390/mcp-web-crawlerstale
An MCP server plugin to crawl all hyperlinks from a website for AI learning purposes.
npm MIT -
simple-web-crawler-scraper-extensiongithub.com/surypratapsingh/simple-web-crawler-scraper-extensionactive
Browser extension + local MCP server giving Claude and other AI agents cheap, read-only eyes on your real browser tabs: read, extract, crawl and export pages locally.
MIT -
A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models.
npm ISC -
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
Apache-2.0 ★ 89 -
The crawler for the agentic era – open-source website crawler & technical SEO auditor for AI agents and humans (macOS, Windows). Rust + Tauri, MCP-native.
Apache-2.0 ★ 47 -
robots.txt generator, validator and checker for AI crawlers (GPTBot, ClaudeBot and more). API + MCP server.
MIT -
fastmcp-web-explorergithub.com/78488393929/fastmcp-web-explorerupdated this year
FastMCP web explorer and scraper for AI agents.
-
MCP server for crawling and indexing web documentation - works with any website
npm MIT -
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
npm MIT ★ 3 -
Web crawling tool that integrates with AI assistants via the MCP
pypi MIT ★ 45 -
Clado Model Context Protocol Server
npm MIT -
Agent-orchestrated web crawler for exact evidence and historical changepoints—with deterministic policy, robots compliance, and local MCP support.
MPL-2.0 ★ 1 -
@iflow-mcp/clado-ai-mcpgithub.com/clado-ai/search-mcpupdated this year
Clado Model Context Protocol Server
npm MIT -
Free website crawler for AI coding agents: any site, JavaScript pages and linked PDFs, into clean text. Skill + MCP server for Claude Code, Codex, Gemini CLI, Cursor, Copilot.
npm MIT -
webcontext-aigithub.com/sumeeth-24/webscrapper-aiupdated this year
Turn any web content into clean AI-ready context — with crawling, chunking, semantic search, and MCP tools
npm MIT -
iflow-mcp_laurentvv-crawl4ai-mcpgithub.com/laurentvv/crawl4ai-mcpupdated this year
A powerful web crawling tool that integrates with AI assistants via the MCP (Machine Conversation Protocol)
pypi MIT -
Web scraping API and hosted MCP server for AI agents. Read public pages as Markdown, HTML, links, or JSON and run bounded crawls with Cloudflare Browser Run.
AGPL-3.0 -
crawlforgegithub.com/makaren-pro/crawlforgeupdated this year
Local-first web-context engine for AI agents with async crawling, BM25 retrieval, token-budgeted context, and MCP.
MIT -
MCP (Model Context Protocol) server for Crawl4AI - Universal web crawling and data extraction
pypi MIT -
Free, open-source SEO crawler and Screaming Frog alternative. Crawl any site locally with no URL limit: technical SEO audit, JavaScript rendering, AI fixes, MCP server.
MIT ★ 1 -
让 AI 代理访问互联网:抓取 · 爬取 · 搜索,内置反爬,$0 无密钥。Web research for AI agents: fetch, crawl & keyless search with anti-bot bypass.
pypi NOASSERTION ★ 8 -
MCP server that gives AI agents the web: fetch, search, crawl and screenshot from one local Rust binary. Zero API keys, zero accounts, Chrome-true TLS.
npm AGPL-3.0-only -
MeshArc MCP server: crawl and scrape websites to clean markdown, and track what changed.
MIT -
Model Context Protocol server for the Pluck web context API: scrape, crawl, search, extract and monitor the web from any MCP client.
npm AGPL-3.0-only -
Self-hosted web scraper for AI agents. Turn any URL into clean Markdown, crawl and map sites, search the web, extract JSON. Python, FastAPI, Playwright, MCP server and CLI.
MIT -
anycrawl-mcpupdated this year
AnyCrawl MCP Server - Adds powerful web scraping and crawling to Cursor, Claude and any other LLM clients
npm MIT -
@leviathangk/playwright-mcpgithub.com/leviathangk/playwright-mcpupdated this year
MCP server for browser automation and web scraping using Playwright. Supports network capture, page understanding, and AI-driven web interaction.
npm MIT -
Verified web evidence and headless browser rendering Model Context Protocol (MCP) server for coding agents
npm MIT -
anycrawl-mcp-serverupdated this year
AnyCrawl MCP Server - Adds powerful web scraping and crawling to Cursor, Claude and any other LLM clients
npm MIT -
Local, unlimited web-search MCP server with BM25 ranking, Playwright crawling, and smart YouTube transcript extraction. DISCOVERY MODE: Get chapter outlines first, then extract specific sections. Perfect for long videos & bug fix workflows. No Docker, no
npm MIT -
bits-biegithub.com/sudharsansm/bieupdated this year
BitSearch Intelligence Engine — real-time, citation-backed web search & extraction for AI apps. Built on Bitscrape.
pypi MIT -
MCP server for sitemap crawling and change detection with LLM-powered filtering - optimized for web scraping workflows
npm MIT -
100% Free & Open-Source Web Crawling MCP Server. Zero-budget, no-API-key web scraping and content extraction for Claude Desktop, Cursor, and AI Agents.
MIT ★ 1 -
MCP server — website awareness for AI agents. Fetch, search, and map sites without raw HTML noise.
npm PolyForm-Noncommercial-1.0.0 -
MCP server for minicrawl: give Claude Desktop, Cursor and other AI agents self-hosted web scraping, crawling, search and AI extraction
npm AGPL-3.0-only -
@iflow-mcp/any4ai-anycrawl-mcp-serverupdated this year
AnyCrawl MCP Server - Adds powerful web scraping and crawling to Cursor, Claude and any other LLM clients
npm MIT -
Fast, stateless CLI for web search, OSS code search, library docs, scraping, and crawling — with an MCP server for agents.
npm MIT -
supacrawlx-mcpgithub.com/SupaCrawlX/supacrawlx-mcpupdated this year
Official SupaCrawlX MCP server — add web scraping, crawling, and video/transcript extraction to Claude, Cursor, and any other LLM client.
MIT -
@autom8minds/seo-mcpgithub.com/autom8minds/seo-mcpupdated this year
MCP server with 25 SEO tools: page analysis, site crawling, security auditing, Core Web Vitals, schema validation, keyword research, backlinks, and more.
npm MIT -
A Model Context Protocol (MCP) server implementation that provides real-time web search capabilities through a simple API
npm ISC