MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
cockroach-crawlergithub.com/ajnasnb/cockroach-crawlerupdated this year
Give AI agents superpowers on the web: crawl, render, map, extract, and verify evidence through one governed Node.js crawler.
npm MIT -
AI layer for graph-crawler: graph search, LLM agents, extraction recipes, MCP and REST servers, change monitoring.
pypi MIT -
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
npm MIT -
Autonomous AI Search, Training & Agent Crawler Diagnostic Engine
npm MIT -
The crawler for the agentic era – open-source website crawler & technical SEO auditor for AI agents and humans (macOS, Windows). Rust + Tauri, MCP-native.
Apache-2.0 ★ 47 -
User-Agent parser API + MCP server: browser, OS and device, plus bot, search crawler, AI crawler and AI agent detection.
MIT -
@crawlready/mcpgithub.com/medartus/crawlready-webupdated this year
CrawlReady MCP server — scan and optimize sites for AI crawlers. Includes agent skills @crawlready-fix and @crawlready-check.
npm MIT -
Native-speed, browserless web crawler + MCP server for AI agents. A standalone Rust engine (turbo-dom + a V8 JS-render tier) — fetch + parse + extract + run page JS with no headless browser. This npm package is a thin launcher that spawns the native binar
npm MIT -
面向应届毕业生的校招/实习信息聚合平台 · 公益项目 · 跳转官方投递入口,不截留简历 | Public-welfare campus recruitment & internship aggregator for Chinese graduates. Next.js + Supabase + Python crawler + MCP for AI agents
MIT -
Free website crawler for AI coding agents: any site, JavaScript pages and linked PDFs, into clean text. Skill + MCP server for Claude Code, Codex, Gemini CLI, Cursor, Copilot.
npm MIT -
See what AI agents do in your Magento 2 store: MCP, UCP and WebMCP calls, AI crawler visits, agent-built orders, bot IP checks and plain-language findings. No IP addresses stored.
MIT ★ 1 -
Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.
npm MIT ★ 266 -
AgentReady REST API and Docker deployment for AI agent readiness scanning. Scan websites, APIs, ecommerce stores, marketplaces, MCP servers and agent services for agentic-web compatibility, discoverability, OpenAPI, llms.txt, structured data, security and AI crawler readiness. Easy to self-host.
Apache-2.0 -
A free AEO audit for your terminal and your agent: what AI crawlers can read on any page, with supereo's free MCP tools. CLI, agent skill and local MCP server.
MIT -
Technical SEO crawler with a local dashboard, an MCP server for agents, and a prioritised AI repair plan
MIT ★ 1 -
Check how ready a website is for AI crawlers, AI search, and AI agents: robots.txt, llms.txt, schema.org JSON-LD, Markdown pages, MCP discovery, and more. CLI, CI, and MCP.
npm MIT -
Run the fastCRW web crawler natively on Windows: tray on/off switch, JS rendering via Chrome, MCP for AI agents, optional sharing over Tailscale.
MIT -
DJAI Academy community-built, free open-source technical SEO crawler and Screaming Frog alternative with JavaScript audits, reports, a local dashboard, and an MCP server for AI agents. Theoretical 100M+ URL architecture.
MIT ★ 7 -
crwgithub.com/fastcrw/crwactive
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
cargo AGPL-3.0 ★ 1,260 -
aeo-mcpgithub.com/arthurcarlsonn/aeo-mcpupdated this year
MCP server for AI visibility: audit crawler permissions, llms.txt, and structured data from any AI agent
npm MIT -
Agent Accessibility Auditorno update data
MCP server for the Mamba Labs Agent Accessibility Auditor actor on Apify. Domain in, whether an AI agent can read that site out: llms.txt, robots AI crawler rules, structured data and endpoints.
smithery -
Evidence-based website scanner for what AI agents can read, answer, and call—validating MCP, WebMCP, APIs, and crawler access.
AGPL-3.0 -
Safe, self-hosted web grounding for AI agents and crawlers — stealth Chrome over MCP
pypi MIT ★ 1 -
MCP server and agent skill for SEO and AI search visibility: audit AI crawler access, generate llms.txt, and build GEO/AEO prompt tests.
npm MIT ★ 2 -
agent-seo-metaactive
Zero-dependency AIEO & GEO (AI Engine Optimization) auditor, AI crawler checker, and Schema.org JSON-LD generator for AI agents and modern web developers.
npm MIT -
AI-native web crawler and structured data extraction engine - one Rust binary with CLI, embedded dashboard, REST API and MCP server for AI agents.
MIT ★ 5 -
seo-audit-mcpgithub.com/cyber-kani/seo-audit-mcpupdated this year
MCP server for fast SEO & GEO checks — audit pages, robots.txt, sitemaps & AI-crawler access from any AI agent. By cloudcoder.dev
MIT -
Open-source, index-free web search engine for AI agents and people. No crawler, no search API bill, no piggybacking on other engines. MCP server for Claude Code, Codex, Cursor and Gemini CLI. Self-hosted.
AGPL-3.0 ★ 2 -
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
cargo Apache-2.0 ★ 274 -
botwire-mcpgithub.com/araspasha/botwire-mcpupdated this year
301 real-time data wires for AI agents. Paid per call via x402 (USDC on Base), no API key, no signup, and no free data tier. Primary sources, not a crawler index: SEC EDGAR, the Federal Reserve, BLS/BEA, federal courts, Congress, DOJ, FDA, CISA/CVEs, USGS
npm MIT -
Free, open-source AI visibility and SEO audit. Crawl your whole site with 643 checks for technical SEO, AEO/GEO, 40 AI crawlers and agentic-web readiness (llms.txt, MCP, A2A), then track whether ChatGPT, Claude, Gemini and Perplexity name you. Fixes for your stack. CLI, dashboard and MCP server; runs locally.
AGPL-3.0 ★ 1 -
Machine Realms (machinerealms.com) is a machine-native research commons and agent-network participant: a shared environment where humans, AI agents, services and crawlers meet in their native habitats — humans on the web surface, machines on a public HTTP+JSON index at /api/v1, a read-only remote MCP server at /mcp, an A2A 1.0 agent card with…
-
SEO & GEO MCP server: Google Search Console, GA4, PageSpeed Insights, structured data, llms.txt, AI crawler checks, WordPress (Yoast, WP-CLI over SSH), GitHub (in-place edits, images) and Gmail attachments as 102 tools for AI agents; optional hosted mode with Google sign-in, per-user tokens and a GitHub App for editing users' own repositories
MIT ★ 30 -
Real-time low-poly 3D action RPG in Godot 4.6 — a 100-floor dungeon crawler built solo. Showcase of the environment art and the pipeline behind it: deterministic voxel carve, rules-based procedural dressing, an in-game sculpt and paint editor, and an MCP server giving an AI agent eyes into headless Godot. Early development.
-
Agent-orchestrated web crawler for exact evidence and historical changepoints—with deterministic policy, robots compliance, and local MCP support.
MPL-2.0 ★ 1 -
Model Context Protocol (MCP) server for AI & LLM crawler readiness via NPX. Audits robots.txt for AI bots, validates /llms.txt, and verifies markdown mirror content negotiation.
npm MIT -
MCP server that crawls GitHub repos and extracts structured data for AI agents
npm MIT -
爬虫多源数据 → 本地 Power BI Desktop 全链路方案(技能 + 82 工具 Power BI MCP 服务端源码):M 清洗流、星型模型与 DAX 规范、PBIP 工程、PBIT 模板导出与跨版本兼容处置、真机刷新验证与逐页截图
-
Website to Clean Markdown, Vector RAG Chunks & llms.txt ($0.50/1k) — Includes 1-Click n8n Workflow, Make.com Blueprint & Claude MCP Server (https://apify.com/lucacelli/ai-ready-website-to-markdown-rag-crawler)
-
simple-web-crawler-scraper-extensiongithub.com/surypratapsingh/simple-web-crawler-scraper-extensionactive
Browser extension + local MCP server giving Claude and other AI agents cheap, read-only eyes on your real browser tabs: read, extract, crawl and export pages locally.
MIT -
actablesite-mcpgithub.com/unitedideas/actablesite-mcpupdated this year
Remote, read-only MCP server for public website readiness and AI crawler-policy audits.
MIT -
A Firecrawl MCP server that rotates multiple API keys for higher throughput and reliability.
MIT ★ 3 -
http-universal-docs-mcpupdated this year
Universal Documentation Crawler MCP Server
npm MIT -
@goldprogrammer/http-universal-docs-mcpupdated this year
Universal Documentation Crawler MCP Server
npm MIT -
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
Apache-2.0 ★ 89 -
mcp-dungeongithub.com/appflavor/mcp-dungeonupdated this year
Text-based dungeon crawler playable by LLM agents via MCP — with a live spectator page and benchmark league
MIT -
AI-powered web crawler — self-hosted Firecrawl alternative. Crawl, extract, render JS, search. CLI + MCP + REST API + Dashboard.
npm Apache-2.0 -
Does blocking GPTBot remove you from ChatGPT? No — OAI-SearchBot is the token that does. Free robots.txt AI-crawler checker (CLI + MCP server) and open data on 165 AI crawlers: which cost you AI answers in ChatGPT, Perplexity and Claude when blocked, and which cost nothing. Includes a 70-store survey.
mcpb MIT -
Fast, low-token web reader and focused crawler for AI assistants, served over MCP. Returns ranked passages instead of whole pages.
cargo Apache-2.0 ★ 7 -
Blazingly fast, local-first website crawler and 120-rule technical SEO audit engine with native MCP support.
Apache-2.0 ★ 6 -
gui-agent-frameworkgithub.com/sjkncs/gui-agent-frameworkupdated this year
Modular block-based GUI automation framework combining VLM vision, LLM planning, Playwright browser control, and pyautogui desktop automation. Includes food review crawler with multi-agent collaboration and MCP server.
-
MCP server — website awareness for AI agents. Fetch, search, and map sites without raw HTML noise.
npm PolyForm-Noncommercial-1.0.0 -
ladon-nousgithub.com/moonyfringers/ladon-nousupdated this year
MCP server exposing Ladon crawl data to AI agents (Claude Desktop, Cursor, etc.).
pypi Apache-2.0 -
BrowserClaw - Control your everyday Chrome browser from AI agents, without losing logins or focus
AGPL-3.0 ★ 41 -
Capture, crawl, and convert documentation websites into Markdown for AI coding agents, CLI workflows, and MCP tools.
npm MIT -
fastmcp-web-explorergithub.com/78488393929/fastmcp-web-explorerupdated this year
FastMCP web explorer and scraper for AI agents.
-
MCP server for XML sitemap discovery, parsing and URL extraction for AI agents
npm MIT -
面向 AI Agent 的微信公众号文章阅读工具。提供 CLI、MCP server 和 Python API。
MIT ★ 18 -
Verified web evidence and headless browser rendering Model Context Protocol (MCP) server for coding agents
npm MIT -
mcp-crawl4aigithub.com/wyattowalsh/mcp-crawl4aiupdated this year
MCP (Model Context Protocol) server wrapping Crawl4AI library as functions using Python
MIT