MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
cockroach-crawlergithub.com/ajnasnb/cockroach-crawlerupdated this year
Give AI agents superpowers on the web: crawl, render, map, extract, and verify evidence through one governed Node.js crawler.
npm MIT -
The MCP server of polite-crawler: nine tools that fetch web pages politely, keep them, search them and verify quotes, for Claude and other MCP clients.
npm MIT -
Fast, low-token web reader and focused crawler for AI assistants, served over MCP. Returns ranked passages instead of whole pages.
Apache-2.0 ★ 7 -
searxNcrawl web crawler with markdown extraction - no DB, no enrichment, just crawling.
pypi MIT -
Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
Apache-2.0 ★ 110 -
MeshArc web crawler for Claude Code: search, scrape, map and crawl sites, and see what changed between crawls.
MIT -
diogenes-web-scraping-engine-mcpgithub.com/stand-out-of-my-sun/diogenes-web-scraping-engine-mcpactive
12 high-performance web extraction, readability & markdown tools: clean LLM markdown compiler, readable article extractor, OpenGraph & JSON-LD metadata parser, HTML table to JSON/Markdown converter, CSS selector isolator, paywall & overlay bypass auditor, XML sitemap crawler, hyperlink extractor, HTML XSS sanitizer, contact & lead info pa
MIT -
Naver Finance web crawler with MCP server support - crawl Korean stock market data
npm MIT -
Web scraper & crawler → clean Markdown with byte-level SHA-256 provenance + WARC archival for LLM/RAG knowledge bases
MIT ★ 2 -
A learning project - a modular web crawler and API built with NestJS, supporting MCP (Model Context Protocol), TypeORM/PostgreSQL, and automatic Swagger documentation
★ 1 -
Agent-orchestrated web crawler for exact evidence and historical changepoints—with deterministic policy, robots compliance, and local MCP support.
MPL-2.0 ★ 1 -
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
npm MIT -
Web crawler server for Open WebUI and MCP, plus a local download CLI, built on crawl4ai
pypi Apache-2.0 -
Safe, self-hosted web grounding for AI agents and crawlers — stealth Chrome over MCP
pypi MIT ★ 1 -
crwgithub.com/fastcrw/crwactive
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
npm AGPL-3.0 ★ 1,259 -
MCP server for technical & on-page SEO audits — crawlability, HTTPS, Core Web Vitals, sitemap, robots.txt, structured data, and AI-crawler access
npm MIT ★ 1 -
MCP server for search and retrieval of web crawler content
pypi Mozilla Public License 2.0 (MPL 2.0) -
MCP server for search and retrieval of web crawler content
pypi MIT License -
MCP server for technical & on-page SEO audits — crawlability, HTTPS, Core Web Vitals, sitemap, robots.txt, structured data, and AI-crawler access. Reports as Markdown, JSON, or a shareable HTML file.
npm MIT -
MCP server tailored to connecting web crawler data and archives
NOASSERTION ★ 46 -
LLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI compatible
MIT ★ 11 -
Web crawler for Open WebUI and MCP clients, plus a download CLI: pages as Markdown, HTML, PDF or screenshots, built on crawl4ai
Apache-2.0 -
acrawl — LLM-powered web crawler. Describe what you want in plain English, get structured data back. Single Rust binary, 25 providers, MCP server built-in.
MIT ★ 16 -
agent-seo-metaactive
Zero-dependency AIEO & GEO (AI Engine Optimization) auditor, AI crawler checker, and Schema.org JSON-LD generator for AI agents and modern web developers.
npm MIT -
Free, local SEO + GEO (AI-search-citation) analysis for AI assistants - audits GEO/AI-crawler access, Core Web Vitals, structured data, security headers, mobile, and images with no signup or API key.
npm FSL-1.1-MIT -
Open-source, index-free web search engine for AI agents and people. No crawler, no search API bill, no piggybacking on other engines. MCP server for Claude Code, Codex, Cursor and Gemini CLI. Self-hosted.
AGPL-3.0 ★ 2 -
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
npm Apache-2.0 ★ 274 -
A local model engine with a search engine and web crawler built in. Run your coding agents on your own GPUs, and get cited answers from your files, code, and the sites you crawl. One executable.
pypi MIT ★ 71 -
Free, open-source AI visibility and SEO audit. Crawl your whole site with 643 checks for technical SEO, AEO/GEO, 40 AI crawlers and agentic-web readiness (llms.txt, MCP, A2A), then track whether ChatGPT, Claude, Gemini and Perplexity name you. Fixes for your stack. CLI, dashboard and MCP server; runs locally.
AGPL-3.0 ★ 1 -
A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models.
npm ISC -
MCP server for the Spider Cloud web crawling and scraping API
MIT ★ 2 -
Crawpapa-Fetchgithub.com/32872-del/Crawpapa-Fetchupdated this year
Agent-oriented crawler analysis MCP server for access diagnostics, page understanding, crawl planning, and structured data extraction.
MIT ★ 1 -
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
Apache-2.0 ★ 89 -
robots.txt generator, validator and checker for AI crawlers (GPTBot, ClaudeBot and more). API + MCP server.
MIT -
The repo's name -_- Also this was written entirely by gemini 2.5 pro
-
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
MIT ★ 3 -
fastmcp-web-explorergithub.com/78488393929/fastmcp-web-explorerupdated this year
FastMCP web explorer and scraper for AI agents.
-
Free website crawler for AI coding agents: any site, JavaScript pages and linked PDFs, into clean text. Skill + MCP server for Claude Code, Codex, Gemini CLI, Cursor, Copilot.
MIT -
Free, open-source SEO crawler and Screaming Frog alternative. Crawl any site locally with no URL limit: technical SEO audit, JavaScript rendering, AI fixes, MCP server.
MIT ★ 1 -
MCP server for web extraction and rendering via Ace Data Cloud.
MIT -
@agenson-horrowitz/web-content-extractor-mcpgithub.com/agenson-tools/web-content-extractor-mcpupdated this year
Agent-optimized MCP server for extracting clean, structured content from web pages - built specifically for AI agents that need LLM-friendly web data
npm MIT -
dlbrowsergithub.com/dekryptlabs/dlbrowserupdated this year
Enterprise-Grade Multi-Backend Browser Engine for AI Agents and Web Automation
pypi MIT -
webcontext-aigithub.com/sumeeth-24/webscrapper-aiupdated this year
Turn any web content into clean AI-ready context — with crawling, chunking, semantic search, and MCP tools
npm MIT -
MCP server that gives AI agents the web: fetch, search, crawl and screenshot from one local Rust binary. Zero API keys, zero accounts, Chrome-true TLS.
npm AGPL-3.0-only -
MCP server for Firecrawl Simple — a web scraping and site mapping tool enabling LLMs to access and process web content
npm MIT -
Bilibili Scraper API (哔哩哔哩 / B站爬虫) — Extract Bilibili videos, XML danmaku bullet comments, user comments with IP geolocation, UP主 creator uploads & Top 100 rankings without cookies or login. Ready for Python, JS, n8n & Claude MCP.
★ 1 -
让 AI 代理访问互联网:抓取 · 爬取 · 搜索,内置反爬,$0 无密钥。Web research for AI agents: fetch, crawl & keyless search with anti-bot bypass.
NOASSERTION ★ 8 -
@mseep/hyperbrowser-mcpgithub.com/hyperbrowserai/mcpupdated this year
Hyperbrowser Model Context Protocol Server
npm ISC -
12 high-performance web extraction, readability & markdown tools: clean LLM markdown compiler, readable article extractor, OpenGraph & JSON-LD metadata parser, HTML table to JSON/Markdown converter, CSS selector isolator, paywall & overlay bypass auditor,
npm MIT -
MeshArc MCP server: crawl and scrape websites to clean markdown, and track what changed.
MIT -
MCP server which collects data from www.xiaohongshu.com social media
GPL-3.0 ★ 4 -
100% Free & Open-Source Web Crawling MCP Server. Zero-budget, no-API-key web scraping and content extraction for Claude Desktop, Cursor, and AI Agents.
MIT ★ 1 -
MCP server for FireCrawl web scraping integration. Supports both cloud and self-hosted instances. Features include web scraping, batch processing, structured data extraction, and LLM-powered content analysis.
npm ISC -
本地搜索引擎 / SEO 引擎 — 多线程全网爬取(DNS 枚举 + 黑名单 + 关键词)、倒排索引、元搜索聚合、统计面板,附带 MCP 服务与 Web 管理端
NOASSERTION ★ 4 -
Fast, stateless CLI for web search, OSS code search, library docs, scraping, and crawling — with an MCP server for agents.
npm MIT -
@iflow-mcp/crawlbase-crawlbase-mcpgithub.com/crawlbase/crawlbase-mcpupdated this year
MCP server for Crawlbase API - enables web scraping through Model Context Protocol
npm MIT -
Fetch any URL to clean markdown for AI agents — Rust fast path, zero-dep MCP server, agent skill. 13W/25T/0L vs Botasaurus on measured corpus (E17).
MIT -
Turn any website into a quality-scored LLM fine-tuning dataset: crawl, chunk, generate Q&A, score, export to JSONL/Parquet/HF Hub. CLI plus MCP server.
MIT ★ 2 -
Web browsing superpowers for AI agents - one MCP server to scrape, crawl, extract, map, search, batch, research & monitor the web. Self-hosted, no API keys. Free Firecrawl alternative.
MIT ★ 25 -
official mcp server for crawlbrulee — EU-native web scraping for AI agents & developers. native scrape and map tools for your agent.
npm Apache-2.0