MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. Each candidate is checked by a language model from its name, description and topics, and only actual MCP servers are listed (not clients, SDKs, lists or skills). Package descriptions, licences and links are read from each package's own registry (npm, PyPI, crates.io, NuGet, RubyGems; Go and Maven packages are only checked to exist), and packages the registry no longer has are removed. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
mcp-as-a-judgegithub.com/OtherVibes/mcp-as-a-judgeupdated this year
MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations
MIT ★ 17 -
Judges agent/RAG outputs (relevance, hallucination, accuracy, routing, prompt injection) using Laya instead of LLM-as-judge -- usable as an MCP server or as a plain Python library.
pypi MIT -
mcp-judgegithub.com/jddavenportOpen/mcp-judgeupdated this year
A calibrated multi-persona LLM-as-judge, exposed as an MCP server and shipped with an eval harness that proves the calibration.
MIT -
Persistent memory + workflow orchestration as an MCP server. 78 canonical tools across 21 namespaces, SQLite-backed, with an LLM-as-judge drift gate and a vibe-flow loop (spec to artifact to drift to publish).
NOASSERTION ★ 2 -
Human-in-the-loop LLM Failure Detection & Benchmark Toolkit — Auto-flag LLM failures (hallucination, refusal, factual error) with LLM-as-Judge explanations. Confirm them in a review UI or via MCP agent — then export a citable benchmark validated against expert labels. Compatible with Ollama and Amazon Bedrock models.
-
A RAG agent that grades its own retrieval, reformulates the query when confidence is low, and abstains instead of hallucinating. LangGraph + hybrid search + reranking, served over MCP, measured against two naive-RAG baselines with an LLM-as-judge harness and RAGAS.
MIT -
RAG-System-from-Scratch-avec-MCPgithub.com/FEDI-HASSINE/RAG-System-from-Scratch-avec-MCPupdated this year
Système RAG complet construit de zéro avec recherche sémantique, reranking, agent IA, API MCP, évaluation LLM-as-a-Judge et interfaces CLI/Streamlit.
★ 1 -
Agentic-Document-Intelligence-Industrialized-Cross-Reference-Enginegithub.com/gayathrikumari/Agentic-Document-Intelligence-Industrialized-Cross-Reference-Engineupdated this year
🤖 Built a production-grade Agentic AI system using MCP + ReAct Multi-Agent Patterns to automate complex engineering document cross-referencing | Multimodal RAG | LLM-as-a-Judge | KV Caching
-
A production-style multi-agent system that generates context-aware briefs. Built on MCP + A2A, with persistent memory (Chroma), dynamic tool routing, self-critique, distributed tracing (Arize Phoenix), an LLM-as-judge eval harness, and Redis caching.
★ 1 -
Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.
MIT ★ 49 -
Guardrails for LLM answers. GroundLens checks an LLM answer against its sources without a model, decides PASS, REVIEW or FAIL under your policy, and keeps a signed record.
Apache-2.0 -
Guardrails for LLM answers. GroundLens checks an LLM answer against its sources without a model, decides PASS, REVIEW or FAIL under your policy, and keeps a signed record.
pypi Apache-2.0 ★ 8 -
JevLab: Offline LLM-as-Judge evaluation, position/verbosity bias diagnostics, Platt calibration & agreement benchmark studio (MCP + CLI + UI)
Apache-2.0 ★ 1 -
The LLM Council works together to answer your hardest questions
MIT ★ 49 -
A blind, sealed LLM-judge panel: the author never grades their own work, and every way to re-roll a grade is refused by code.
Apache-2.0 -
Benchmark LLMs on your own scenarios with a multi-judge panel. Full MCP server (tools, resources, prompts, progress) plus local GUI, CLI, and desktop app. One OpenRouter key.
MIT -
Let your coding agent fix a prompt until it actually works, and prove it did. A self-hosted MCP server: cross-vendor judges, frozen baselines and memory.
Apache-2.0 ★ 10 -
atlasgithub.com/2005-Aneeshdutt/atlasupdated this year
Automated LLM red-teaming pipeline: a LangGraph agent runs jailbreak, prompt-injection, hallucination and bias attacks against a model, scores responses with an LLM judge, mutates failed attacks, and reports results in a Streamlit dashboard and MCP agent.
MIT ★ 1 -
Is your LLM judge a coin flip? One command tests it for position bias, flip-flopping and padding bias, no labels needed, then turns your real logs into an eval suite.
MIT -
Ask several LLMs the same question, have them grade each other blind, and measure how biased the graders are. A provider-agnostic LLM Council with judge-bias analytics, CLI, MCP server and web UI.
MIT ★ 1 -
DeeWaanAI — local-first MCP server and visual pipeline designer. Runs entirely on your machine; your keys, your data, your models.
npm AGPL-3.0 -
llmswapgithub.com/sreenathmmenon/llmswapupdated this year
Ask, compare, and synthesize across 10 LLM providers. Python SDK, CLI, local Arena, MCP, and privacy-aware
pypi MIT ★ 36 -
errorbar MCP server — every platform API (evals, criteria, gates, logs, datasets, aliases, audit, proving) as tools for Claude, Cursor, and any MCP client.
npm Apache-2.0 -
poc_scrapergithub.com/pjnhek/poc_scraperupdated this year
Grounded GTM account research: ICP scoring and citation-traced outreach, exposed as a live hosted MCP server. Every claim traces to a numbered retrieval or gets dropped.
MIT -
Self-hosted multi-agent AI platform: generates, evaluates and refines LLM answers until they hit a quality bar (~22% more accurate, ~45% fewer hallucinations)
MIT -
정부 R&D 매뉴얼 5권을 인쇄 쪽번호 인용과 함께 답하는 RAG. 하이브리드 검색, 6방식 벤치마크, 로컬 MCP로 Claude Desktop 배포
-
quorumgithub.com/danielhansenjones/quorumupdated this year
Multi-agent RAG system on LangGraph: parallel per-axis analyst agents over hybrid retrieval (Qdrant, BGE-M3) and Postgres, a tool-using critic for claim-level fact-checking, MCP server, and an eval harness with A/B campaigns and a QLoRA-distilled local judge.
MIT -
Test each claim in a thesis against a measurement and get MATCH, DRIFT or UNVERIFIABLE, with a sealed record anyone can recheck offline. No model in the verdict step.
NOASSERTION ★ 3 -
The agent that turns a pile of context into a defensible comparison: one MCP tool surface, two runtimes, span-level citations and honest uncertainty. Produces comparanda analyses.
Apache-2.0 -
A Swiss Army knife that lets AI agents get the most out of decision models, simply. Typed yes/no, choice and score judgments over MCP, REST and a CLI, to gate and triage agent work. Written in Rust.
MIT ★ 2 -
Jev MCP server: a decision layer for coding agents, built on TypeSafe Jev (System One model). Ship gates, risk checks, file triage that keeps files out of context, and a safe headless browser, with calibrated confidence. For Claude Code, Codex, Cursor.
npm MIT ★ 6 -
fusion-mcpgithub.com/yksanjo/fusion-mcpupdated this year
Zero-dep MCP server exposing OpenRouter Fusion (multi-model panel + judge synthesis) as a tool any agent can call.
MIT -
Turn AI coding CLIs (Claude Code, Codex, Antigravity) into AI employees on LINE, Telegram, Discord, Slack and 7 more chat apps. A judge reviews each result, every cost is logged, memory stays on your machine. Also runs as a standalone MCP memory server. Self-hosted, Apache 2.0.
Apache-2.0 ★ 51