MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
MCP server for LLM evaluation gates.
pypi MIT -
llm-eval-searchgithub.com/fvahedian/llm-eval-searchupdated this year
—
-
Benchmark that scores LLMs on fixing Arma 3 SQF bugs through three MCP tools; the traps and the results table stay server-side.
MIT -
Open source AI evaluation framework — LLM-as-judge + assertion-based evals for any AI app. CLI + MCP server.
npm Apache-2.0 -
Governance-classifier eval harness on the Claude Messages API, with calibrated LLM-as-judge and an MCP server
MIT -
An LLM eval harness graded by computed ground truth, not an LLM judge. Worked example: poker, served to models as MCP tools — gpt-4o-mini goes from 26/75 to 75/75 with them.
MIT ★ 1 -
Is your LLM judge a coin flip? One command tests it for position bias, flip-flopping and padding bias, no labels needed, then turns your real logs into an eval suite.
MIT -
frontkitgithub.com/lakshmi930/frontkitupdated this year
MCP server + eval gate for four small AI tools. Deterministic checks and an LLM judge fail the build when model behaviour drifts.
-
Eval-driven SQL reliability for AI agents — generate, validate, safely execute, and regression-test LLM SQL. MCP + FastAPI gateways, execution-truth evals, CI regression gate.
pypi MIT ★ 3 -
The plug-and-play evals layer for LLM-based projects: library, learning center, check center, API, MCP server.
MIT ★ 1 -
LLM observability and evals in a single binary: traces, prompts, datasets, scores and review queues for LLM and agent apps. Self-hosted, SQLite inside, OpenTelemetry-native, MCP and CLI for coding agents.
go Apache-2.0 ★ 1 -
Jira sprint metrics computed by code, reported by an LLM, guarded against invented numbers. MCP server, evals and tracing included.
MIT -
Self-hosted LLM operations gateway in one Go binary: OpenAI-compatible router, durable agent tracker, faithfulness evals + drift, traces, MCP server, console
Apache-2.0 ★ 2 -
Python framework + local Studio for reliable LLM workflows: typed files in your repo, pre-run checks, experiments and evals, MCP server for coding agents
pypi NOASSERTION ★ 4 -
MCP server that three-way-matches supplier invoices against purchase orders: deterministic rule engine, golden-case evals, tests. LLM as interface, rules as policy.
MIT -
agents-AIgithub.com/RenanMiqueloti/agents-AIupdated this year
Agentes LangGraph com HITL via interrupt(), MCP server custom e evals LLM-as-judge — multi-provider (Claude, OpenAI, Ollama)
MIT -
Multi-agent RAG on LangGraph: agents research, write and fact-check cited daily notes. Hybrid search (Postgres full-text + pgvector), an LLM router with fallbacks, evals in CI, an MCP server. FastAPI, Astro.
Apache-2.0 -
An autonomous LLM fund that scores its own stock predictions — Brier score, calibration curves, eval + ablation harness — and serves its full decision history over MCP. LangGraph pipeline with deterministic risk guardrails.
★ 1 -
Local meeting note-taker for Windows: records calls without a bot, transcribes offline (faster-whisper), writes notes with a local LLM (Ollama). Evals, tests, MCP server.
MIT -
PokéDex con IA: agente con tool calling escrito a mano, servidor MCP, 3 LLMs con fallback (Groq → Gemini → Anthropic), evals y visión. Next.js + Supabase (RLS) + FastAPI.
MIT -
Payment routing engine: route by empirical approval evidence (Wilson LB), retry by error class, cost_bias knob; LLM only at the edges (decline normalizer with evals) + MCP server. Live demo + API on Vercel.
MIT -
Telegram assistant for a nail studio: MCP tools over the studio API, writes only after human confirmation, streaming replies and per-chat memory. Model chosen with an 87-case eval across 6 LLMs.
-
LLM agent that authors and runs integration tests for a complex 3-process C payments backend through a 15-tool MCP server, grounded in a generated knowledge base, with a controlled eval measuring whether the KB helps.
MIT ★ 1 -
Enterprise agentic IT/Ops support triage: a multi-step LLM agent with an MCP server, RBAC + human-approval gates, hash-chained audit, RAG with data governance, and an eval-gated CI pipeline. Python/FastAPI + React/TS; runs fully offline.
MIT -
MCP server exposing live F1 game UDP telemetry to LLM agents, with evals (TypeScript)
MIT -
mcp-judgegithub.com/jddavenportOpen/mcp-judgeupdated this year
A calibrated multi-persona LLM-as-judge, exposed as an MCP server and shipped with an eval harness that proves the calibration.
MIT -
LLM tools on a 690k-job dataset: MCP server, agent evals, salary extraction with held-out tests
MIT -
evalbenchgithub.com/cognis-digital/evalbenchupdated this year
Offline LLM / agent eval harness with regression gates
NOASSERTION -
code-review-crewgithub.com/siddlpskmi/code-review-crewupdated this year
Autonomous multi-agent AI code reviewer with custom MCP server, agentic RAG, and LLM-as-judge evals
-
An agentic music librarian with a deterministic validator, a repair loop, and a measured eval suite. LLM tool use, RAG, tracing and MCP.
MIT -
research-pilotgithub.com/Marwaguerfel/research-pilotupdated this year
Autonomous AI research agent: parallel planning, RAG memory, MCP server, LLM-as-judge eval, Streamlit UI
-
Ask a SQLite database questions in plain language. Local LLM agent with tools, evals and an MCP server.
MIT ★ 1 -
Multi-tenant Sinhala/English RAG assistant with an AI agent, MCP server and LLM evals in CI.
NOASSERTION -
LLM planning agent over a forecasting backtest — strict tools, structured output, measured eval with failure analysis
-
Double-entry ledger with Kafka fraud detection and a local-LLM investigation agent (LangGraph + MCP), with an eval harness
MIT -
Autonomous Prompt Engineering, Jailbreak Defense, LLM Cost Optimizer & Eval Benchmark MCP for Cursor, Claude & Windsurf
-
RAG support agent for banking security FAQs — LangGraph, ChromaDB, MCP, LLM-judged faithfulness eval in CI (16/16, p95 1.9s)
-
LLM agent and MCP server that audits model code for look-ahead and selection bias, scored on a planted-bug eval suite.
MIT -
MCP servers that let LLM agents operate Juju safely: server-side policy guardrails, audit log, Prometheus-driven autoscaling, and an eval harness for open-weight models.
Apache-2.0 -
LLM agent for business websites: answers from a knowledge base, captures leads · OpenTelemetry GenAI tracing, evals, MCP server · FastAPI · RU/EN/AR
-
Observability MCP server for AI agents - 89 tools to query LLM cost/errors/latency across 7 providers and operate alerts, budgets, evals from Claude/Cursor
npm MIT -
Domain-pluggable LLM agent runtime with tool permissions, human approval, MCP server, and a repeatable eval harness. Local models (llama.cpp) and hosted APIs.
MIT -
Autonomous game playtesting agent: plays a game, hunts bugs with generic oracles + LLM, writes reproducible reports. Includes seeded-bug eval harness and MCP server.
MIT -
Testing AI-facing APIs: an own MCP server and an LLM endpoint on Claude with evals, groundedness and prompt-injection checks
MIT -
Multi-agent LLM red-teaming and eval lab: attacks a RAG chatbot with prompt injection, poisoned documents and PII probes, scores replies via an MCP server (Presidio, DeepEval), and logs every run to SQLite for prompt and model comparison.
-
Give your AI agents an evaluation tool - provider-agnostic LLM eval runner with an MCP server for Claude, structured output validation, datasets, evaluators and honest cost metrics
pypi MIT -
Answers RFPs from your own product docs with cited, quality-checked drafts. Go API, Python workers, pgvector, Redis Streams, MCP server, LLM eval harness.
MIT ★ 1 -
Model Context Protocol (MCP) server exposing workflow-shaped tools to Claude/LLMs — held-out eval in CI (Node 20+22), stderr-safe logging, security-noted README.
MIT -
Multi-tenant B2B call analytics platform — LangGraph pipeline, MCP server, pgvector RAG, JWT auth with Postgres Row-Level Security, and automated LLM-as-judge eval harness
-
inbox-mcpgithub.com/jairo21991-debug/inbox-mcpupdated this year
MCP server exposing a CRM-style inbox (contact search, conversation retrieval, BM25 full-text search) to LLM agents — with a retrieval eval harness (Hit@k/Recall@k/MRR).
MIT -
industrial-data-analyst-agentgithub.com/MohamedAdelAbdelrahman/industrial-data-analyst-agentactive
LLM agent for industrial sensor data: LangGraph, own MCP server, fine-tuned Text-to-SQL (QLoRA), RAG, sandboxed Python, eval
-
Negotiate with an AI car dealer that must keep its floor price secret. Three LLM-security levels (ADK, LangGraph, MCP, A2A) measured with evals, deployed on Cloud Run.
★ 1 -
An MCP server that gives any LLM host grounded, cited retrieval over your own documents — hybrid BM25 + dense retrieval, cross-encoder-ready reranking, and built-in eval.
npm MIT -
MCP-native AIOps incident intelligence: PySpark + Postgres, XGBoost, a cited LangGraph agent on a local LLM, evals with a CI gate, dbt, Kafka, Airflow, Helm/Kind
-
MCP server that lets an LLM read chats, voice notes and images and draft replies, behind a tested send gate and a 15-case red-team eval.
MIT ★ 1 -
HomelabSentinel — an AI SRE for my homelab that asks before it acts: LangGraph agent with a human-approval gate, MCP server, BM25 RAG, evals and Langfuse tracing on a 100% local LLM.
MIT -
Evidence-grounded IT support agent: Gemini function calling over an MCP-style tool server (KB search, ticket search, ticket drafts), with a 60-question eval harness, LLM judge, and OpenTelemetry tracing.
★ 1 -
A from-scratch Model Context Protocol server (stdio, stdlib-only, zero deps): safe calc (OWASP LLM06), BM25 search, a no-LLM-judge answer grader, and live model-drift / eval-run lookups. 42 tests.
MIT ★ 1 -
Key-free MCP servers that let LLMs build real TouchDesigner networks: local knowledge base (no API keys), offline .toe/.tox generation, live TD editing. Eval-gated - 636/636 build-token-exact.
AGPL-3.0 ★ 1 -
Run LLM eval suites, cache every call, and compare two runs item by item against a noise floor. CLI and MCP server.
pypi MIT