MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
Autonomous AI Search, Training & Agent Crawler Diagnostic Engine
npm MIT -
AI layer for graph-crawler: graph search, LLM agents, extraction recipes, MCP and REST servers, change monitoring.
pypi MIT -
AI crawler checker: Chrome extension + MCP server that shows which content GPTBot, ClaudeBot and other AI crawlers can't see because it needs JavaScript (raw HTML vs rendered).
npm AGPL-3.0 -
Website to Clean Markdown, Vector RAG Chunks & llms.txt ($0.50/1k) — Includes 1-Click n8n Workflow, Make.com Blueprint & Claude MCP Server (https://apify.com/lucacelli/ai-ready-website-to-markdown-rag-crawler)
-
cockroach-crawlergithub.com/ajnasnb/cockroach-crawlerupdated this year
Give AI agents superpowers on the web: crawl, render, map, extract, and verify evidence through one governed Node.js crawler.
npm MIT -
Model Context Protocol (MCP) server for AI & LLM crawler readiness via NPX. Audits robots.txt for AI bots, validates /llms.txt, and verifies markdown mirror content negotiation.
npm MIT -
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
npm MIT -
MCP server for Site Crawler for Links and Tables. Crawl a site and get links, download links, HTML tables as rows, and lists. No code, no AI. Runs on Apify. Clay-ready output.
npm MIT -
Give your AI a real map of the codebase. Code-Crawler turns a project into a queryable property graph of files, types, methods, annotations, calls, virtual dispatch, and aspect advice — then exposes it to Claude, Cursor, Copilot, and any MCP client as precise tools.
npm MIT -
The crawler for the agentic era – open-source website crawler & technical SEO auditor for AI agents and humans (macOS, Windows). Rust + Tauri, MCP-native.
Apache-2.0 ★ 45 -
robots.txt generator, validator and checker for AI crawlers (GPTBot, ClaudeBot and more). API + MCP server.
MIT -
actablesite-mcpgithub.com/unitedideas/actablesite-mcpupdated this year
Remote, read-only MCP server for public website readiness and AI crawler-policy audits.
MIT -
Local-first MCP server for AI visibility audits: GEO/AEO, AI SEO, llms.txt, schema, sitemap and AI crawler readiness.
npm MIT ★ 2 -
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
npm MIT ★ 3 -
User-Agent parser API + MCP server: browser, OS and device, plus bot, search crawler, AI crawler and AI agent detection.
MIT -
crawlio-mcpgithub.com/crawlio-app/crawlioupdated this year
MCP server for Crawlio — control the macOS website crawler from AI assistants
npm MIT -
Free, open-source SEO crawler and Screaming Frog alternative. Crawl any site locally with no URL limit: technical SEO audit, JavaScript rendering, AI fixes, MCP server.
MIT ★ 1 -
@crawlready/mcpgithub.com/medartus/crawlready-webupdated this year
CrawlReady MCP server — scan and optimize sites for AI crawlers. Includes agent skills @crawlready-fix and @crawlready-check.
npm MIT -
Native-speed, browserless web crawler + MCP server for AI agents. A standalone Rust engine (turbo-dom + a V8 JS-render tier) — fetch + parse + extract + run page JS with no headless browser. This npm package is a thin launcher that spawns the native binar
npm MIT -
Free open-source SEO crawler CLI with AI-search (GEO/AEO) readiness checks, HTML reports, and an MCP server.
MIT -
dungeon-mcpgithub.com/jianengdev/dungeon-mcpupdated this year
AI-narrated dungeon crawler. MCP server runs the rules, FastAPI streams play over WebSocket, Next.js renders chat + map.
-
AI-powered web crawler — self-hosted Firecrawl alternative. Crawl, extract, render JS, search. CLI + MCP + REST API + Dashboard.
npm Apache-2.0 -
GEO readiness MCP server: checks robots.txt AI-crawler access, schema, llms.txt, sitemap. MIT.
MIT -
Fast, low-token web reader and focused crawler for AI assistants, served over MCP. Returns ranked passages instead of whole pages.
cargo Apache-2.0 ★ 7 -
Technical SEO crawler with a local dashboard, an MCP server for agents, and a prioritised AI repair plan
MIT ★ 1 -
A local MCP server for QueryRev AI Readiness: AI crawler access, robots.txt and llms.txt checks, as tools. No model calls, no telemetry.
npm MIT -
ESLint for AI search — audit AI crawler access, llms.txt, structured data and citability for ChatGPT, Perplexity, Claude & co.
npm MIT ★ 4 -
mcp-crawl-paritygithub.com/arturseo-geo/mcp-crawl-parityupdated this year
MCP server for Googlebot vs AI crawler parity analysis — compare crawl behaviour from Nginx logs with GSC data to detect visibility gaps
MIT ★ 1 -
面向应届毕业生的校招/实习信息聚合平台 · 公益项目 · 跳转官方投递入口,不截留简历 | Public-welfare campus recruitment & internship aggregator for Chinese graduates. Next.js + Supabase + Python crawler + MCP for AI agents
MIT -
Run the fastCRW web crawler natively on Windows: tray on/off switch, JS rendering via Chrome, MCP for AI agents, optional sharing over Tailscale.
MIT -
Does blocking GPTBot remove you from ChatGPT? No — OAI-SearchBot is the token that does. Free robots.txt AI-crawler checker (CLI + MCP server) and open data on 165 AI crawlers: which cost you AI answers in ChatGPT, Perplexity and Claude when blocked, and which cost nothing. Includes a 70-store survey.
mcpb MIT -
Free website crawler for AI coding agents: any site, JavaScript pages and linked PDFs, into clean text. Skill + MCP server for Claude Code, Codex, Gemini CLI, Cursor, Copilot.
npm MIT -
Check how ready a website is for AI crawlers, AI search, and AI agents: robots.txt, llms.txt, schema.org JSON-LD, Markdown pages, MCP discovery, and more. CLI, CI, and MCP.
npm MIT -
Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.
npm MIT ★ 266 -
MCP server that checks whether a website is set up for AI the way its owner intends: can GPTBot, ClaudeBot, PerplexityBot and 50+ other AI crawlers read it, and what does its access policy allow? Generates llms.txt, an AI-crawler robots.txt and JSON-LD, reads your Answer Analysis, and checks whether a page permits an AI use.
npm MIT -
A free AEO audit for your terminal and your agent: what AI crawlers can read on any page, with supereo's free MCP tools. CLI, agent skill and local MCP server.
MIT -
Free, open-source desktop SEO crawler for Windows and macOS: technical SEO audits, JavaScript rendering, structured data, accessibility, Search Console/GA4/PageSpeed, built-in AI embeddings, CLI and MCP server. Rust + Tauri.
Apache-2.0 -
See what AI agents do in your Magento 2 store: MCP, UCP and WebMCP calls, AI crawler visits, agent-built orders, bot IP checks and plain-language findings. No IP addresses stored.
MIT ★ 1 -
AgentReady REST API and Docker deployment for AI agent readiness scanning. Scan websites, APIs, ecommerce stores, marketplaces, MCP servers and agent services for agentic-web compatibility, discoverability, OpenAPI, llms.txt, structured data, security and AI crawler readiness. Easy to self-host.
Apache-2.0 -
MCP server for SEO and GEO (Generative Engine Optimization) audits — on-page SEO, structured data, robots.txt, sitemaps, hreflang, and AI-crawler access for ChatGPT, Claude, Perplexity and Gemini. No API keys required.
npm MIT ★ 1 -
Free cookieless WordPress analytics plugin: real-time visitor stats with humans, bots and AI crawlers counted separately, outbound click tracking, public Stats API and MCP server. Free alternative to Plausible, Fathom, Matomo, MonsterInsights and Google Analytics.
-
16-bit dark-fantasy 2D Pixel-Art Dungeon Crawler RPG in Godot 4.4 & Unity 6 (C#) with a 5-room procedural dungeon, Tactical AI Auto-Pilot, Multi-Phase Bosses, 1,000-match Monte Carlo Combat Simulator, and Bi-Directional MCP/UDP Server.
MIT -
Free, open-source CLI and MCP server that checks whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot) and search engines can access your website: robots.txt, llms.txt, sitemap, JSON-LD, meta tags, JavaScript rendering and Common Crawl presence. Runs locally.
npm MIT -
Is your site ready for AI search? Open-source SEO, AEO & GEO audits: AI crawler access, llms.txt, answer-ready pages, AI visibility, and an AI analyst's to-do list. CLI, GitHub Action, Claude Code plugin, MCP server.
MIT ★ 5 -
DJAI Academy community-built, free open-source technical SEO crawler and Screaming Frog alternative with JavaScript audits, reports, a local dashboard, and an MCP server for AI agents. Theoretical 100M+ URL architecture.
MIT ★ 7 -
crwgithub.com/fastcrw/crwactive
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
cargo AGPL-3.0 ★ 1,260 -
Your search data, kept. Self-hosted SEO, GEO and ASO analytics in your own Cloudflare account: Search Console, Bing, AI crawlers, plus App Store Connect and Google Play store funnels, search terms, ratings and reviews. Ask through MCP or use the built-in dashboard.
MIT ★ 1 -
Check how crawlers see a site, send new pages to search engines and AI crawlers, see crawl receipts. Official remote MCP server.
-
AI-visibility audit: AI crawler access, identity files, page signals, hosted analysis.
npm MIT -
Look up domain reputation snapshots and AI crawler policy from Domduck's public store.
npm MIT -
AI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.
npm MIT -
aeo-mcpgithub.com/arthurcarlsonn/aeo-mcpupdated this year
MCP server for AI visibility: audit crawler permissions, llms.txt, and structured data from any AI agent
npm MIT -
Open-source GEO & SEO readiness probe for AI crawler access, llms.txt, structured data, sitemaps, and performance.
oci MIT -
fal-ai-model-databasegithub.com/PHY041/fal-ai-model-databaseupdated this year
🚀 Complete database of 1,094 fal.ai models with pricing, categories & API docs. Includes Python crawler, MCP server for Claude, and exportable JSON/CSV data.
MIT ★ 10 -
store-checks-mcpgithub.com/Nightwolf7570/store-checks-mcpupdated this year
MCP server: check any site's AI-crawler robots policy + JSON-LD product completeness. Stdlib-only Python.
MIT -
AI-powered web crawler that captures designs and generates GSAP animation code (~$0.10/site). Built with Playwright, Claude AI, and MCP for seamless integration with Claude CLI.
★ 1 -
Interactive OSINT Lead-Chaining Caseboard, Target Dossier Workbench, Wayback Machine Crawler & BYOK AI Detective
MIT -
Agent Accessibility Auditorno update data
MCP server for the Mamba Labs Agent Accessibility Auditor actor on Apify. Domain in, whether an AI agent can read that site out: llms.txt, robots AI crawler rules, structured data and endpoints.
smithery -
MCP server that inspects a website's AI-search readiness: AI crawler access, llms.txt, schema markup, and indexing directives
npm MIT -
Official WebAudits.pro Model Context Protocol (MCP) server for DOM bloat, AI crawler GEO readiness, and LCP diagnostics.
MIT ★ 1