MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
cockroach-crawlergithub.com/ajnasnb/cockroach-crawlerupdated this year
Give AI agents superpowers on the web: crawl, render, map, extract, and verify evidence through one governed Node.js crawler.
npm MIT -
A MCP-based documentation crawler service
npm MIT -
The MCP server of polite-crawler: nine tools that fetch web pages politely, keep them, search them and verify quotes, for Claude and other MCP clients.
npm MIT -
AI crawler checker: Chrome extension + MCP server that shows which content GPTBot, ClaudeBot and other AI crawlers can't see because it needs JavaScript (raw HTML vs rendered).
AGPL-3.0 -
Agent-orchestrated web crawler for exact evidence and historical changepoints—with deterministic policy, robots compliance, and local MCP support.
MPL-2.0 ★ 1 -
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
npm MIT -
A command-line tool acting as an MCP (ModelContextProtocol) server, using Playwright to crawl web content for AI models.
npm ISC -
A simple Model Context Protocol (MCP) server for crawling Tor onion sites and extracting their content.
MIT ★ 3 -
Model Context Protocol (MCP) server for AI & LLM crawler readiness via NPX. Audits robots.txt for AI bots, validates /llms.txt, and verifies markdown mirror content negotiation.
npm MIT -
A learning project - a modular web crawler and API built with NestJS, supporting MCP (Model Context Protocol), TypeORM/PostgreSQL, and automatic Swagger documentation
★ 1 -
A Wikipedia Image Search Tool. Follows Creative Commons Licences for images and uses them in your projects via Claude Desktop/Cline.
★ 11 -
Live in-process debugger, input bot and MCP server for PC games: Unreal reflection, memory, hooks, breakpoints and a hot-reload C++ plugin API
MIT -
Website to Clean Markdown, Vector RAG Chunks & llms.txt ($0.50/1k) — Includes 1-Click n8n Workflow, Make.com Blueprint & Claude MCP Server (https://apify.com/lucacelli/ai-ready-website-to-markdown-rag-crawler)
-
Crawpapa-Fetchgithub.com/32872-del/Crawpapa-Fetchupdated this year
Agent-oriented crawler analysis MCP server for access diagnostics, page understanding, crawl planning, and structured data extraction.
MIT ★ 1 -
robots.txt generator, validator and checker for AI crawlers (GPTBot, ClaudeBot and more). API + MCP server.
MIT -
actablesite-mcpgithub.com/unitedideas/actablesite-mcpupdated this year
Remote, read-only MCP server for public website readiness and AI crawler-policy audits.
MIT -
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
MIT ★ 3 -
Local-first MCP server for AI visibility audits: GEO/AEO, AI SEO, llms.txt, schema, sitemap and AI crawler readiness.
MIT ★ 2 -
AI crawler checker MCP server: shows which page content GPTBot and ClaudeBot miss because it needs JavaScript. Raw HTML vs rendered DOM, from the Unrendered Chrome extension.
npm AGPL-3.0-only -
Free, open-source SEO crawler and Screaming Frog alternative. Crawl any site locally with no URL limit: technical SEO audit, JavaScript rendering, AI fixes, MCP server.
MIT ★ 1 -
User-Agent parser API + MCP server: browser, OS and device, plus bot, search crawler, AI crawler and AI agent detection.
MIT -
search_itgithub.com/quantumxiaol/search_itupdated this year
A MCP Server of Search and Crawler
MIT -
AI-powered JMComic crawler with MCP server and skills management for seamless AI agent integration
pypi MIT -
searxNcrawl web crawler with markdown extraction - no DB, no enrichment, just crawling.
pypi MIT -
Naver Finance web crawler with MCP server support - crawl Korean stock market data
npm MIT -
Web crawler server for Open WebUI and MCP, plus a local download CLI, built on crawl4ai
pypi Apache-2.0 -
Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
Apache-2.0 ★ 110 -
MeshArc web crawler for Claude Code: search, scrape, map and crawl sites, and see what changed between crawls.
MIT -
GEO readiness MCP server: checks robots.txt AI-crawler access, schema, llms.txt, sitemap. MIT.
MIT -
Fast, low-token web reader and focused crawler for AI assistants, served over MCP. Returns ranked passages instead of whole pages.
Apache-2.0 ★ 7 -
Web scraper & crawler → clean Markdown with byte-level SHA-256 provenance + WARC archival for LLM/RAG knowledge bases
MIT ★ 2 -
mcp-crawl-paritygithub.com/arturseo-geo/mcp-crawl-parityupdated this year
MCP server for Googlebot vs AI crawler parity analysis — compare crawl behaviour from Nginx logs with GSC data to detect visibility gaps
MIT ★ 1 -
ESLint for AI search — audit AI crawler access, llms.txt, structured data and citability for ChatGPT, Perplexity, Claude & co.
npm MIT ★ 4 -
Does blocking GPTBot remove you from ChatGPT? No — OAI-SearchBot is the token that does. Free robots.txt AI-crawler checker (CLI + MCP server) and open data on 165 AI crawlers: which cost you AI answers in ChatGPT, Perplexity and Claude when blocked, and which cost nothing. Includes a 70-store survey.
MIT -
Free website crawler for AI coding agents: any site, JavaScript pages and linked PDFs, into clean text. Skill + MCP server for Claude Code, Codex, Gemini CLI, Cursor, Copilot.
MIT -
MCP server that checks whether a website is set up for AI the way its owner intends: can GPTBot, ClaudeBot, PerplexityBot and 50+ other AI crawlers read it, and what does its access policy allow? Generates llms.txt, an AI-crawler robots.txt and JSON-LD, reads your Answer Analysis, and checks whether a page permits an AI use.
MIT -
A free AEO audit for your terminal and your agent: what AI crawlers can read on any page, with supereo's free MCP tools. CLI, agent skill and local MCP server.
MIT -
Free cookieless WordPress analytics plugin: real-time visitor stats with humans, bots and AI crawlers counted separately, outbound click tracking, public Stats API and MCP server. Free alternative to Plausible, Fathom, Matomo, MonsterInsights and Google Analytics.
-
Free, open-source CLI and MCP server that checks whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot) and search engines can access your website: robots.txt, llms.txt, sitemap, JSON-LD, meta tags, JavaScript rendering and Common Crawl presence. Runs locally.
npm MIT -
DJAI Academy community-built, free open-source technical SEO crawler and Screaming Frog alternative with JavaScript audits, reports, a local dashboard, and an MCP server for AI agents. Theoretical 100M+ URL architecture.
MIT ★ 7 -
diogenes-web-scraping-engine-mcpgithub.com/stand-out-of-my-sun/diogenes-web-scraping-engine-mcpactive
12 high-performance web extraction, readability & markdown tools: clean LLM markdown compiler, readable article extractor, OpenGraph & JSON-LD metadata parser, HTML table to JSON/Markdown converter, CSS selector isolator, paywall & overlay bypass auditor, XML sitemap crawler, hyperlink extractor, HTML XSS sanitizer, contact & lead info pa
MIT -
Is your site ready for AI search? Open-source SEO, AEO & GEO audits: AI crawler access, llms.txt, answer-ready pages, AI visibility, and an AI analyst's to-do list. CLI, GitHub Action, Claude Code plugin, MCP server.
MIT ★ 5 -
MCP server for Crawl Readiness: check what AI crawlers can read on a site and what its access policy allows, check whether a page permits an AI use, generate llms.txt, robots.txt and JSON-LD, and read your Answer Analysis.
npm MIT -
crwgithub.com/fastcrw/crwactive
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
npm AGPL-3.0 ★ 1,259 -
Your search data, kept. Self-hosted SEO, GEO and ASO analytics in your own Cloudflare account: Search Console, Bing, AI crawlers, plus App Store Connect and Google Play store funnels, search terms, ratings and reviews. Ask through MCP or use the built-in dashboard.
MIT ★ 1 -
Check how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts. Official remote MCP server.
-
MCP server for search and retrieval of web crawler content
pypi Mozilla Public License 2.0 (MPL 2.0) -
MCP server for search and retrieval of web crawler content
pypi MIT License -
mcp-crew-riskstale
This is a Crawler Security Guard based on the Model Context Protocol (MCP).
npm MIT -
mcp-marketing-sitestale
This is a Crawler Security Guard based on the Model Context Protocol (MCP).
npm MIT -
mcp-system-infrastale
This is a Crawler Security Guard based on the Model Context Protocol (MCP).
npm MIT -
@iflow-mcp/mcp-crew-riskstale
This is a Crawler Security Guard based on the Model Context Protocol (MCP).
npm MIT -
A sovereign, local-first MCP server for AI search crawler governance, robots.txt auditing, brand safety guardrails, and IndexNow notifications.
pypi MIT -
MCP server tailored to connecting web crawler data and archives
NOASSERTION ★ 46 -
store-checks-mcpgithub.com/Nightwolf7570/store-checks-mcpupdated this year
MCP server: check any site's AI-crawler robots policy + JSON-LD product completeness. Stdlib-only Python.
MIT -
Interactive OSINT Lead-Chaining Caseboard, Target Dossier Workbench, Wayback Machine Crawler & BYOK AI Detective
MIT -
Agent Accessibility Auditorno update data
MCP server for the Mamba Labs Agent Accessibility Auditor actor on Apify. Domain in, whether an AI agent can read that site out: llms.txt, robots AI crawler rules, structured data and endpoints.
smithery -
MCP server for technical & on-page SEO audits — crawlability, HTTPS, Core Web Vitals, sitemap, robots.txt, structured data, and AI-crawler access
npm MIT ★ 1 -
LLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI compatible
MIT ★ 11 -
Audit websites with this self-hosted SEO crawler. Connect LibreCrawl to Claude and perform unlimited technical audits without per-crawl fees.