MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
Web crawler for datatist.com
pypi MIT -
—
-
mcpp-crawlerupdated this year
MCP 网页爬虫工具 - 抓取网页内容、提取结构化数据
npm -
mcp-universal-crawlergithub.com/litvan007/mcp-universal-crawlerupdated this year
—
MIT -
RSS Crawler MCP Server
★ 19 -
SearchProcurementTenders-crawler.Vergithub.com/h30190/SearchProcurementTenders-crawler.Verupdated this year
一個用來查標案的mcp
MIT ★ 15 -
cockroach-crawlergithub.com/ajnasnb/cockroach-crawlerupdated this year
Give AI agents superpowers on the web: crawl, render, map, extract, and verify evidence through one governed Node.js crawler.
npm MIT -
Web Crawler MCP
npm MIT -
mcp-crawlergithub.com/Tai-DT/mcp-crawlerupdated this year
MCP Crawler - Web scraping and data extraction engine
★ 3 -
A MCP-based documentation crawler service
npm MIT -
High-performance filesystem crawler MCP server powered by Rust
npm MIT -
mcp-universal-crawlergithub.com/yywl07/mcp-universal-crawlerupdated this year
A universal image crawler tool using MCP protocol and SSE.
MIT ★ 1 -
Web crawler MCP server for extracting text content from web pages
npm MIT -
threads-mcp-crawlergithub.com/dkyazzentwatwa/threads-mcp-crawlerupdated this year
Mark Zuckerberg's worst nightmare. Crawls Meta Threads for data.
MIT ★ 10 -
Web crawler MCP server for extracting text content from web pages
npm MIT -
一个功能强大的多平台自媒体数据采集MCP工具,支持B站、小红书、抖音、快手、抖音微博、贴吧、知乎等主流平台的公开信息抓取。
★ 80 -
amz-crawler-mcpupdated this year
MCP Server for extracting Amazon A+ Content template preview HTML/CSS
npm MIT -
erica-crawler-mcpupdated this year
MCP server for crawling raw HTML from URLs with hash authentication
npm MIT -
Model Context Protocol (MCP) server that provides web crawling functionality.
MIT -
Autonomous AI Search, Training & Agent Crawler Diagnostic Engine
npm MIT -
AI layer for graph-crawler: graph search, LLM agents, extraction recipes, MCP and REST servers, change monitoring.
pypi MIT -
MCP server that crawls GitHub repos and extracts structured data for AI agents
npm MIT -
@houtini/seo-crawler-mcpgithub.com/houtini-ai/seo-crawler-mcpupdated this year
Crawl and analyse websites for SEO errors and issues using Crawlee with SQLite storage
npm Apache-2.0 -
smart-crawler-mcpgithub.com/mguozhen/smart-crawlerupdated this year
Install helper for the smart-crawler Agent-first MCP server (Claude, Codex, Cursor).
npm MIT -
@cd39390/mcp-web-crawlerstale
An MCP server plugin to crawl all hyperlinks from a website for AI learning purposes.
npm MIT -
daum-news-crawler-jyupdated this year
MCP server that crawls Daum news headlines for a given keyword and saves them to a text file
pypi MIT -
mcp-issue-crawlersgithub.com/CNnx686/mcp-issue-crawlersupdated this year
It can retrieve MCP server(different) code and issues of the mcp via GitHub API
-
Site-wide crawler MCP server: BFS crawl of all internal links with time/title/URL filters, page operations (summarize/click/download), SQLite persistence. Built on crawl4ai.
MIT -
AI crawler checker: Chrome extension + MCP server that shows which content GPTBot, ClaudeBot and other AI crawlers can't see because it needs JavaScript (raw HTML vs rendered).
npm AGPL-3.0 -
A Model Context Protocol (MCP) server providing ethical web crawling and search capabilities
npm MIT -
A simple Model Context Protocol (MCP) server for crawling Tor onion sites and extracting their content.
MIT ★ 3 -
Agent-orchestrated web crawler for exact evidence and historical changepoints—with deterministic policy, robots compliance, and local MCP support.
MPL-2.0 ★ 1 -
dedalus-marketplace-crawler-tsgithub.com/BUZDOLAPCI/dedalus-marketplace-crawler-tsupdated this year
A Model Context Protocol server that crawls the Dedalus Marketplace to discover and list all available MCP servers with their metadata
★ 1 -
A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models.
npm ISC -
Adaptive web crawler — only launches a browser when a page actually needs one. LLM-optimized markdown, multi-engine search, a 32-tool MCP server, and autonomous research + tracking. TypeScript.
MIT -
MCP server for crawling Korean stock and company information from Naver Finance with support for both HTTP and stdio transports
npm MIT -
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
npm MIT -
Model Context Protocol (MCP) server for AI & LLM crawler readiness via NPX. Audits robots.txt for AI bots, validates /llms.txt, and verifies markdown mirror content negotiation.
npm MIT -
A Wikipedia Image Search Tool. Follows Creative Commons Licences for images and uses them in your projects via Claude Desktop/Cline.
★ 11 -
A powerful Model Context Protocol (MCP) server for crawling websites and extracting assets including images and SEO metadata. Built for e-commerce sites and general web crawling needs.
MIT ★ 6 -
A learning project - a modular web crawler and API built with NestJS, supporting MCP (Model Context Protocol), TypeORM/PostgreSQL, and automatic Swagger documentation
★ 1 -
MCP server for Site Crawler for Links and Tables. Crawl a site and get links, download links, HTML tables as rows, and lists. No code, no AI. Runs on Apify. Clay-ready output.
npm MIT -
simple-web-crawler-scraper-extensiongithub.com/surypratapsingh/simple-web-crawler-scraper-extensionactive
Browser extension + local MCP server giving Claude and other AI agents cheap, read-only eyes on your real browser tabs: read, extract, crawl and export pages locally.
MIT -
A Model Context Protocol (MCP) server for web crawling and news extraction. Enables LLMs to browse the web, extract article content, and process news feeds using newspaper4k and modern web scraping techniques.
MIT ★ 1 -
Give your AI a real map of the codebase. Code-Crawler turns a project into a queryable property graph of files, types, methods, annotations, calls, virtual dispatch, and aspect advice — then exposes it to Claude, Cursor, Copilot, and any MCP client as precise tools.
npm MIT -
Credentialed web-data pipeline (X, Reddit, newswires, SEC EDGAR) with an MCP interface Claude queries directly. CDP session reuse, per-site adapters, dedup store.
★ 1 -
Live in-process debugger, input bot and MCP server for PC games: Unreal reflection, memory, hooks, breakpoints and a hot-reload C++ plugin API
MIT -
爬虫多源数据 → 本地 Power BI Desktop 全链路方案(技能 + 82 工具 Power BI MCP 服务端源码):M 清洗流、星型模型与 DAX 规范、PBIP 工程、PBIT 模板导出与跨版本兼容处置、真机刷新验证与逐页截图
-
Scrapling 风格隐身爬虫库:自适应选择器 · TLS 指纹 HTTP · JS 渲染 · 回调式 Spider 框架(重试/robots/去重/中间件管道)· JS 逆向 Agent · 验证码识别 · MCP 服务 · 本地 Web UI · 授权渗透侦察 · 默认 SSRF 防护
MIT ★ 9 -
Website to Clean Markdown, Vector RAG Chunks & llms.txt ($0.50/1k) — Includes 1-Click n8n Workflow, Make.com Blueprint & Claude MCP Server (https://apify.com/lucacelli/ai-ready-website-to-markdown-rag-crawler)
-
The crawler for the agentic era – open-source website crawler & technical SEO auditor for AI agents and humans (macOS, Windows). Rust + Tauri, MCP-native.
Apache-2.0 ★ 45 -
Crawpapa-Fetchgithub.com/32872-del/Crawpapa-Fetchupdated this year
Agent-oriented crawler analysis MCP server for access diagnostics, page understanding, crawl planning, and structured data extraction.
MIT ★ 1 -
robots.txt generator, validator and checker for AI crawlers (GPTBot, ClaudeBot and more). API + MCP server.
MIT -
actablesite-mcpgithub.com/unitedideas/actablesite-mcpupdated this year
Remote, read-only MCP server for public website readiness and AI crawler-policy audits.
MIT -
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
npm MIT ★ 3 -
Local-first MCP server for AI visibility audits: GEO/AEO, AI SEO, llms.txt, schema, sitemap and AI crawler readiness.
npm MIT ★ 2 -
search_itgithub.com/quantumxiaol/search_itupdated this year
A MCP Server of Search and Crawler
MIT -
iflow-mcp_noimank-fnewscrawlergithub.com/noimank/fnewscrawlerupdated this year
Financial news and data crawler with MCP protocol support
pypi MIT -
MCP (Model Context Protocol) server exposing Spider web crawler capabilities as tools.
cargo MIT -
Free, open-source SEO crawler and Screaming Frog alternative. Crawl any site locally with no URL limit: technical SEO audit, JavaScript rendering, AI fixes, MCP server.
MIT ★ 1