MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
mcp-crawl-paritygithub.com/arturseo-geo/mcp-crawl-parityupdated this year
MCP server for Googlebot vs AI crawler parity analysis — compare crawl behaviour from Nginx logs with GSC data to detect visibility gaps
MIT ★ 1 -
Naver Finance web crawler with MCP server support - crawl Korean stock market data
npm MIT -
claude-browser-crawl-cdpgithub.com/gruncode/claude-browser-crawl-cdpupdated this year
Drive a real, logged-in Chrome from Claude Code over CDP — a persistent Playwright REPL, ~25x faster web crawling & automation.
MIT -
hotelrate-crawlgithub.com/seanbabalala/hotelrate-crawlupdated this year
Real-time hotel price comparison across Trip.com, Booking.com, Agoda & Google Hotels — MCP server for AI agents (Claude, Cursor, etc.)
MIT ★ 12 -
MCP server for Crawl Readiness: check what AI crawlers can read on a site and what its access policy allows, check whether a page permits an AI use, generate llms.txt, robots.txt and JSON-LD, and read your Answer Analysis.
npm MIT -
DJAI Academy community-built, free open-source technical SEO crawler and Screaming Frog alternative with JavaScript audits, reports, a local dashboard, and an MCP server for AI agents. Theoretical 100M+ URL architecture.
MIT ★ 7 -
MCP server that checks whether a website is set up for AI the way its owner intends: can GPTBot, ClaudeBot, PerplexityBot and 50+ other AI crawlers read it, and what does its access policy allow? Generates llms.txt, an AI-crawler robots.txt and JSON-LD, reads your Answer Analysis, and checks whether a page permits an AI use.
MIT -
crawlemoongithub.com/razavioo/crawlemoonupdated this year
Advanced Web Crawling Platform with Deep Analysis and MCP Server
pypi MIT -
exa-mcpgithub.com/2aronS/exa-mcpupdated this year
MCP server for web search and crawling using Exa API
MIT ★ 6 -
Exa MCP for web search and web crawling!
MIT ★ 5,234 -
A Model Context Protocol (MCP) server that provides search and crawl functionality using Search1API
npm MIT -
Official Spidra MCP server: web search, scraping, crawling, document parsing, templates and monitors for AI agents
npm MIT -
anti-botgithub.com/TaroYamada08/anti-botupdated this year
Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls. 🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
BSD-3-Clause ★ 1 -
webcontext-aigithub.com/sumeeth-24/webscrapper-aiupdated this year
Turn any web content into clean AI-ready context — with crawling, chunking, semantic search, and MCP tools
npm MIT -
Spider MCP server — crawl, scrape, search, and automate browsers at scale. 100K+ pages/sec with anti-bot bypass.
npm MIT -
claude-code-mcpsgithub.com/arturseo-geo/claude-code-mcpsupdated this year
5 MCP servers for Claude Code — SEO page auditor, SERP intelligence, Common Crawl backlinks, advanced GSC, WordPress management. By TheGEOLab.net
MIT ★ 2 -
@iflow-mcp/deep-research-mcpgithub.com/pinkpixel-dev/deep-research-mcpupdated this year
A Model Context Protocol (MCP) server for performing deep web research using Tavily Search and Crawl APIs, preparing structured data for LLM-powered documentation generation.
npm MIT -
The focused-crawling MCP server for AI agents. Rank a whole site by your topic, read any page as clean Markdown, extract anything
MIT ★ 12 -
web-research-assistantgithub.com/elad12390/web-research-assistantupdated this year
Comprehensive MCP server for web research with 13 tools, 4 resources, and 5 prompts: search, crawl, package info, GitHub stats, error translation, API docs discovery, and more.
pypi MIT -
MeshArc web crawler for Claude Code: search, scrape, map and crawl sites, and see what changed between crawls.
MIT -
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
Apache-2.0 ★ 89 -
Headless MCP server for Screaming Frog SEO Spider – run crawls, export and analyze crawl data via the CLI. Small locked-down tool surface for scheduled audits, CI, and unattended AI agents.
pypi MIT ★ 96 -
A self-hosted alternative to Screaming Frog: your crawl, your machine, no per-URL pricing and no seats. 222 SEO checks — 180 of them proven to fire against a site built to be wrong. JS rendering, crawl diffing with stable issue ids, axe-core accessibility, server logs, Search Console, an HTTP API and an MCP server.
NOASSERTION ★ 11 -
TechCyclopedia: High-Fidelity Agent Data Pipeline for Technical Documentation Crawling
MIT -
@iflow-mcp/mcp-crawl4ai-tsgithub.com/omgwtfwow/mcp-crawl4ai-tsupdated this year
TypeScript MCP server for Crawl4AI - web crawling and content extraction
npm MIT -
A simple Model Context Protocol (MCP) server for crawling Tor onion sites and extracting their content.
MIT ★ 3 -
TypeScript implementation of an MCP server for Crawl4AI. Provides tools for web crawling, content extraction, and browser automation.
MIT ★ 14 -
Crawpapa-Fetchgithub.com/32872-del/Crawpapa-Fetchupdated this year
Agent-oriented crawler analysis MCP server for access diagnostics, page understanding, crawl planning, and structured data extraction.
MIT ★ 1 -
searxNcrawl web crawler with markdown extraction - no DB, no enrichment, just crawling.
pypi MIT -
TypeScript MCP server that gives coding agents Tavily Search, Extract, Crawl and Research.
-
MCP server for the Spider Cloud web crawling and scraping API
MIT ★ 2 -
Free Rust SEO and GEO toolkit powered by TypeSafe Jev: 64-rule audits, site crawls, AI citation checks, rank drift, CI gates, 19-tool MCP. Zero cost.
npm MIT -
web-searchgithub.com/Chi-9966/web-searchupdated this year
MCP server for multi-engine web search, fetch, crawl & vision — Baidu / Bing / DuckDuckGo / SearXNG
MIT ★ 1 -
mcp-webscrapergithub.com/samirsaci/mcp-webscraperupdated this year
Local MCP Server with Claude Desktop (Windows + WSL) with scrapping and crawling tools.
★ 4 -
Compare web scraping, crawling and SERP API pricing and limits against your workload and budget
-
Check how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts. Official remote MCP server.
-
Zed extension for Olostep, web data infrastructure for AI. Live web search, scraping and crawling API to extract and structure data.
MIT -
Bing Webmaster Tools & IndexNow API MCP server for AI agents: search performance, crawl issues, and instant URL indexing.
pypi MIT -
Cleanroom website recreation engine — headless multi-page crawl, vision-driven React 19 synthesis, and 3-tier visual verification.
npm MIT ★ 1 -
Agentic web infrastructure: fetch, scrape, extract, download with MCP server, browser fingerprinting, and crawl-scrape-download pipeline
MIT ★ 2 -
MCP server (stdio) for Bing Webmaster Tools: query/page stats, sitemaps, URL submission + IndexNow, crawl diagnostics, keyword & link research
npm MIT -
MCP server for ApexApi — call 120+ AI models and live web context (scrape/crawl/extract) from any MCP client with one ak- API key
npm MIT -
Fast, stateless CLI for web search, OSS code search, library docs, scraping, and crawling — with an MCP server for agents.
npm MIT -
apexapi-mcpgithub.com/apexapi/apexapi-mcpupdated this year
MCP server for ApexApi — 120+ AI models plus live web context (scrape/crawl/extract) from any MCP client, one prepaid ak- key
MIT -
Analyse robots.txt content and report whether a path may be crawled.
npm MIT -
MeshArc MCP server: crawl and scrape websites to clean markdown, and track what changed.
MIT -
Crawl corporate domains for verified emails, LinkedIn company profiles, Twitter/X, and executive contacts. MCP Server & Python SDK.
-
Industrial cleanroom website recreation engine — headless multi-page crawl, vision-driven React synthesis, and 3-tier verification
npm MIT -
MCP server + CLI for AI-native SEO audits: broken links, internal link suggestions, crawl diffs, optional local embeddings.
MIT ★ 2 -
API-based web scraping MCP server for AI agents. 20 tools for scrape, crawl, search, and extract. Apify actor: https://apify.com/red.cars/firecrawl-mcp
-
A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models.
npm ISC -
让 AI 代理访问互联网:抓取 · 爬取 · 搜索,内置反爬,$0 无密钥。Web research for AI agents: fetch, crawl & keyless search with anti-bot bypass.
pypi MIT License -
Open-source, self-hosted Tavily alternative: web search, extract, crawl and map API for AI agents, with an MCP server
MIT ★ 3 -
Audit websites with this self-hosted SEO crawler. Connect LibreCrawl to Claude and perform unlimited technical audits without per-crawl fees.
-
Crawl, index, and hybrid-search documentation sites; served to LLM agents as an MCP server. FastAPI + DuckDB + BM25/embeddings, fully offline-testable.
MIT -
让 AI 代理访问互联网:抓取 · 爬取 · 搜索,内置反爬,$0 无密钥。Web research for AI agents: fetch, crawl & keyless search with anti-bot bypass.
NOASSERTION ★ 8 -
This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.
MIT ★ 44 -
Open-source Cloudflare Browser Rendering proxy — 10 MCP tools for Claude Code (content, screenshot, PDF, markdown, scrape, JSON AI extraction, links, a11y, crawl)
MIT ★ 9 -
Search, scrape, crawl, and answer with sources, from your own machine. A CLI and MCP server with one optional key: Jev.
Apache-2.0 ★ 3 -
domscout browser API as MCP tools: screenshots, Markdown extraction, structured data, DOM snapshots, page automation, and crawls.
MIT