MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
GEO readiness MCP server: checks robots.txt AI-crawler access, schema, llms.txt, sitemap. MIT.
MIT -
A local MCP server for QueryRev AI Readiness: AI crawler access, robots.txt and llms.txt checks, as tools. No model calls, no telemetry.
npm MIT -
Model Context Protocol (MCP) server for AI & LLM crawler readiness via NPX. Audits robots.txt for AI bots, validates /llms.txt, and verifies markdown mirror content negotiation.
npm MIT -
context-cligithub.com/hanselhansel/context-cliupdated this year
LLM readiness linter for websites. Audits robots.txt, llms.txt, Schema.org, and content density on a 0-100 scale. Includes MCP server. Published on PyPI: pip install context-cli.
MIT ★ 5 -
Model Context Protocol (MCP) server & CLI for GEO, AEO, LLMs.txt, AI Robots.txt, and JSON-LD schema audits for Generative Search & AI Agents.
npm MIT -
Crawler/SEO companion for Dash apps: /llms.txt, /robots.txt, /sitemap.xml, bot detection, static-HTML prerender, and an MCP bridge. Works with Flask, FastAPI, and Quart backends (Dash 4.1+).
pypi MIT -
Can ChatGPT, Claude and Perplexity read and cite your site? Checks robots.txt, edge blocks (fetches as each bot), llms.txt, schema, JS-only content. Zero dependencies. Also an MCP server.
pypi MIT -
truesource-geo-mcpgithub.com/saschadeforth/truesource-geo-mcpupdated this year
Make any website visible to ChatGPT, Claude, Gemini & Perplexity. Free MCP tools for AI-readiness scoring, robots.txt/llms.txt generation, VibeTags, and AI bot analysis. No API key required.
npm MIT -
robots.txt generator, validator and checker for AI crawlers (GPTBot, ClaudeBot and more). API + MCP server.
MIT -
Is this website ready for AI agents? MCP server + client for the AgentReady audit: robots.txt rules for 17 AI bots (GPTBot, ClaudeBot, ChatGPT-User, PerplexityBot...), bot challenges, content in raw HTML, markdown, llms.txt, sitemap, JSON-LD, Open Graph,
npm MIT -
mcp-seogithub.com/gmoneyn/mcp-seoupdated this year
SEO toolkit MCP server — analyze meta tags, robots.txt, sitemaps, keyword density, readability, and heading structure
npm MIT -
Check how ready a website is for AI crawlers, AI search, and AI agents: robots.txt, llms.txt, schema.org JSON-LD, Markdown pages, MCP discovery, and more. CLI, CI, and MCP.
npm MIT -
seo-toolkit-mcpgithub.com/gmoneyn/seo-toolkit-mcpupdated this year
SEO toolkit MCP server — analyze meta tags, robots.txt, sitemaps, keyword density, readability, and heading structure
npm MIT -
PriceMoved: pricing-page change monitor (waitlist stage) with a remote MCP server for product info and a robots.txt check.
MIT -
MCP server for SEO and GEO (Generative Engine Optimization) audits — on-page SEO, structured data, robots.txt, sitemaps, hreflang, and AI-crawler access for ChatGPT, Claude, Perplexity and Gemini. No API keys required.
npm MIT ★ 1 -
Free, open-source CLI and MCP server that checks whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot) and search engines can access your website: robots.txt, llms.txt, sitemap, JSON-LD, meta tags, JavaScript rendering and Common Crawl presence. Runs locally.
npm MIT -
A polite browser for AI agents: live sessions with refs, stale-ref and irreversible-action refusals, markdown reading, robots.txt and pacing, bring-your-own proxies, site skills, and an MCP server.
Apache-2.0 -
Agent skill, MCP server and x402 client for Online Web Directory: check how a website treats AI agents (bot wall, JavaScript, robots, llms.txt) before you fetch it. Free lookups, or 1 cent per call over x402 with no key.
MIT-0 -
@thomasgorisse/seo-mcpgithub.com/thomasgorisse/seo-mcpupdated this year
MCP server for SEO optimization — meta tag generator, Schema.org markup, sitemap builder, robots.txt generator, Open Graph tags, keyword density analyzer, and page speed checklist.
npm Apache-2.0 -
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
npm MIT -
Honest web search and page reading for AI agents — an MCP server and CLI that identify themselves truthfully, respect robots.txt, and hand bot-checks to a human in a visible browser. Headless until it matters.
npm MIT -
Maximize site visibility for AI engines — llms.txt, Markdown endpoints, robots.txt AI directives, AI discovery files, and MCP abilities.
GPL-2.0 ★ 1 -
AI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.
npm MIT -
INITE Agents for WordPress: llms.txt, /.well-known agent files, robots.txt Content-Signal, WebMCP and MCP Adapter abilities. Read-only mirror; issues go to inite-ai/inite-agents.
GPL-2.0 -
@inite/site-agents: a Next.js site serves its INITE-confirmed facts to AI agents (llms.txt, .well-known, robots.txt, WebMCP, JSON-LD). Read-only mirror; issues go to inite-ai/inite-agents.
npm MIT -
@mntglobal/agentreadygithub.com/mnt-global/agentreadyupdated this year
Is your store invisible to AI shopping agents? Scan any store for agent-readiness: schema.org, feeds, robots/llms.txt, ACP/UCP/MCP discovery.
npm MIT -
mcp-fetchgithub.com/LangGPT/mcp-fetchupdated this year
A Model Context Protocol server for web content fetching without robots.txt restrictions
pypi MIT ★ 2 -
Checks a site build: sitemaps, structured data, robots.txt and what crawlers see.
npm -
Audit whether an AI agent can read a site: llms.txt, robots AI rules, structured data.
npm MIT -
Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
npm -
mcp-seo-auditgithub.com/mk-techi/mcp-seo-auditupdated this year
MCP server for on-page SEO analysis audit any URL's metas, schema, robots.txt, sitemaps and links
npm MIT -
MCP server for fetching, crawling, and analyzing websites. 8 tools: fetch, crawl, extract, SEO audit, sitemap, broken links, screenshots. Anti-bot protections, throttling, robots.txt compliance.
-
@leeseokwoon/mcp-websearchgithub.com/yourusername/mcp-websearchupdated this year
Real-time web search MCP server with no robots.txt restrictions
npm MIT -
mcp-web-fetch-langgptstale
A Model Context Protocol server providing tools to fetch and convert web content for usage by LLMs (robots.txt checking removed)
pypi MIT -
mcp-geogithub.com/josezuma/mcp-geoupdated this year
MCP server exposing GEO tools to any AI agent: check_llms_txt, audit_robots_for_ai, validate_schema, geo_audit.
MIT -
dsh-webfetchgithub.com/withlovehub/dsh-webfetchupdated this year
Zero-dependency webfetch MCP server for DeepSeek Harness and any MCP client: clean text / markdown / raw HTML / JSON, robots.txt compliance, chunked reading, SSRF protection.
MIT ★ 2 -
MCP server that checks whether a website is set up for AI the way its owner intends: can GPTBot, ClaudeBot, PerplexityBot and 50+ other AI crawlers read it, and what does its access policy allow? Generates llms.txt, an AI-crawler robots.txt and JSON-LD, reads your Answer Analysis, and checks whether a page permits an AI use.
npm MIT -
SEO toolkit as an MCP server — free, keyless on-page audits, robots.txt, sitemaps, structured data, broken-link checks, and PageSpeed insights
MIT ★ 6 -
Does blocking GPTBot remove you from ChatGPT? No — OAI-SearchBot is the token that does. Free robots.txt AI-crawler checker (CLI + MCP server) and open data on 165 AI crawlers: which cost you AI answers in ChatGPT, Perplexity and Claude when blocked, and which cost nothing. Includes a 70-store survey.
mcpb MIT -
MCP server that turns any public web page into clean Markdown, respecting robots.txt. Free tier, then pay per call via x402.
MIT -
Analyse robots.txt content and report whether a path may be crawled.
npm MIT -
Agent Accessibility Auditorno update data
MCP server for the Mamba Labs Agent Accessibility Auditor actor on Apify. Domain in, whether an AI agent can read that site out: llms.txt, robots AI crawler rules, structured data and endpoints.
smithery -
Comprehensive HTTP fetch MCP server with SSRF protection, HTML-to-markdown, and robots.txt awareness
npm MIT ★ 2 -
MCP server for technical & on-page SEO audits — crawlability, HTTPS, Core Web Vitals, sitemap, robots.txt, structured data, and AI-crawler access
npm MIT ★ 1 -
webmesh-mcpgithub.com/creatorpiyush/webmesh-mcpupdated this year
Token-economical web browsing, scraping, and crawling MCP server. Tiered fetching (static → browser), robots.txt compliance, session persistence, and BFS crawling.
npm MIT -
mcp-web-agentgithub.com/creatorpiyush/mcp-web-agentupdated this year
Token-economical web browsing, scraping, and crawling MCP server. Tiered fetching (static → browser), robots.txt compliance, session persistence, and BFS crawling.
npm MIT -
@codetidy/mcpupdated this year
MCP server for CodeTidy developer tools — JSON formatting, Base64, URL encoding, hashing, semver, CORS, robots.txt, and 55+ more. Free, runs locally.
npm MIT -
AgentReady (T6 CLI edition) — audit a public storefront for AI shopping-agent readiness from CI: UCP profile, live MCP endpoint, llms.txt, robots AI policy, Product structured data, sitemap, platform, HTTPS/canonical.
pypi MIT -
Airbnb Search and Listing Details Serverno update data
Search Airbnb listings and retrieve detailed information about specific listings. Get structured JSON data without requiring an API key, while respecting Airbnb's robots.txt rules. Simplify your vacation rental planning with easy access to Airbnb data.
smithery -
seo-audit-mcpgithub.com/cyber-kani/seo-audit-mcpupdated this year
MCP server for fast SEO & GEO checks — audit pages, robots.txt, sitemaps & AI-crawler access from any AI agent. By cloudcoder.dev
MIT -
MCP server, skill and Claude plugin for SEO and competitor research. Reads sitemaps and feeds into tables, robots.txt respected.
npm Apache-2.0 ★ 1 -
mcp-seo-progithub.com/gmoneyn/mcp-seo-proupdated this year
Pro SEO toolkit MCP server — 12 tools for meta tags, robots.txt, sitemaps, keyword density, readability, heading structure, JSON-LD validation, OG debugging, internal links, page speed, competitor analysis, and content scoring
npm MIT -
meok-w3c-tdm-rights-mcpgithub.com/csoai-org/meok-w3c-tdm-rights-mcpupdated this year
MEOK W3C TDM Rights MCP — EU CDSM Directive Article 4(3) Text + Data Mining opt-out. Issues + scans HTTP/HTML/robots.txt/C2PA reservation signals. Liability shield for AI training pipelines. By MEOK AI Labs.
pypi MIT -
An SEO MCP server (stdio CLI). Free, keyless on-page & technical SEO tools — single-page and site-wide audits, robots.txt, sitemap, structured-data, broken-link and keyword checks, professional PDF reports, plus PageSpeed/Lighthouse (Google PSI or local h
npm MIT -
Pro SEO toolkit MCP server — 12 tools for meta tags, robots.txt, sitemaps, keyword density, readability, heading structure, JSON-LD validation, OG debugging, internal links, page speed, competitor analysis, and content scoring
npm SEE LICENSE IN LICENSE -
mcp-web-scrapegithub.com/mukul975/mcp-web-scrapeupdated this year
🚀 mcp-web-scrape — Clean, cache-aware web content fetcher for AI agents. Fetch any URL → extract readable content → return Markdown/JSON with citations. ⚡ Fast caching, 🤝 robots.txt compliant, 📝 Markdown-ready output, �� works with ChatGPT/Claude Desktop.
npm MIT ★ 50 -
MCP server for AI agents to read any web page in full — past the robots.txt and bot blocks that stop native fetch — as clean, ad-stripped Markdown. Plus web search.
pypi Apache-2.0 -
Web pages and PDFs to clean, LLM-ready Markdown as a remote MCP tool (Claude, Cursor, VS Code) and via the Apify API. Token counts, small crawls, sitemaps, RAG chunks, respects robots.txt. Python and Node examples.
MIT -
URL to Markdown for AI agents and LLMs: turn any public web page into clean Markdown (HTML to Markdown, SSRF-safe, robots.txt-aware). MCP server (stdio + remote), zero-dependency library and x402 pay-per-call API ($0.002 USDC on Base).
npm MIT -
Autonomous AI Search, Training & Agent Crawler Diagnostic Engine
npm MIT