MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. Each candidate is checked by a language model from its name, description and topics, and only actual MCP servers are listed (not clients, SDKs, lists or skills). Package descriptions, licences and links are read from each package's own registry (npm, PyPI, crates.io, NuGet, RubyGems; Go and Maven packages are only checked to exist). Packages are found through ecosyste.ms; its metadata (CC BY-SA 4.0) is kept only where the registry could not be read, and packages the registry no longer has are removed. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
llm-as-judge-mcp-servergithub.com/jgsheppa/llm-as-judge-mcp-serverupdated this year
An MCP server to judge an LLM's responses
npm MIT -
judge-audit-mcpgithub.com/asif786ka/judge-audit-mcpupdated this year
Judge QA for agents — measures whether your LLM judge agrees with humans, drifted between runs, or is biased, before you ship the number
pypi MIT -
LLM-as-Judge 텍스트 품질 평가를 MCP 서버로 — 루브릭 채점·골든셋 시험·교차 검증
MIT ★ 1 -
Local Laya System-1 judge MCP server + skill for oh-my-pi: typed decisions in milliseconds, zero LLM tokens
NOASSERTION ★ 13 -
mcp-as-a-judgegithub.com/OtherVibes/mcp-as-a-judgeupdated this year
MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations
pypi MIT ★ 17 -
Judges agent/RAG outputs (relevance, hallucination, accuracy, routing, prompt injection) using Laya instead of LLM-as-judge -- usable as an MCP server or as a plain Python library.
pypi MIT -
mcp-judgegithub.com/jddavenportOpen/mcp-judgeupdated this year
A calibrated multi-persona LLM-as-judge, exposed as an MCP server and shipped with an eval harness that proves the calibration.
MIT -
contrariangithub.com/anthonymicho/contrarianupdated this year
MCP server adding an independent second LLM perspective (Judge B) with full project graph context
pypi AGPL-3.0-or-later -
Governance-classifier eval harness on the Claude Messages API, with calibrated LLM-as-judge and an MCP server
MIT -
企业级智能客服 AI 系统参考实现:三路意图融合 · MCP 工具调用 · 三级记忆 · LangGraph 多 Agent 编排 · Monitor 闭环 · LLM-as-Judge 评测。零密钥可运行,含两引擎对照数据与诚实记录。
MIT ★ 3 -
MCP server for automated insurance claim processing — LLM-as-judge pattern, Agent Skills, TypeScript
★ 1 -
JevLab: Offline LLM-as-Judge evaluation, position/verbosity bias diagnostics, Platt calibration & agreement benchmark studio (MCP + CLI + UI)
Apache-2.0 ★ 1 -
A blind, sealed LLM-judge panel: the author never grades their own work, and every way to re-roll a grade is refused by code.
Apache-2.0 -
Ask several LLMs the same question, have them grade each other blind, and measure how biased the graders are. A provider-agnostic LLM Council with judge-bias analytics, CLI, MCP server and web UI.
MIT ★ 1 -
Benchmark LLMs on your own scenarios with a multi-judge panel. Full MCP server (tools, resources, prompts, progress) plus local GUI, CLI, and desktop app. One OpenRouter key.
MIT -
atlasgithub.com/2005-Aneeshdutt/atlasupdated this year
Automated LLM red-teaming pipeline: a LangGraph agent runs jailbreak, prompt-injection, hallucination and bias attacks against a model, scores responses with an LLM judge, mutates failed attacks, and reports results in a Streamlit dashboard and MCP agent.
MIT ★ 1 -
Is your LLM judge a coin flip? One command tests it for position bias, flip-flopping and padding bias, no labels needed, then turns your real logs into an eval suite.
MIT -
Human-in-the-loop LLM Failure Detection & Benchmark Toolkit — Auto-flag LLM failures (hallucination, refusal, factual error) with LLM-as-Judge explanations. Confirm them in a review UI or via MCP agent — then export a citable benchmark validated against expert labels. Compatible with Ollama and Amazon Bedrock models.
-
Persistent memory + workflow orchestration as an MCP server. 78 canonical tools across 21 namespaces, SQLite-backed, with an LLM-as-judge drift gate and a vibe-flow loop (spec to artifact to drift to publish).
npm NOASSERTION ★ 2 -
A RAG agent that grades its own retrieval, reformulates the query when confidence is low, and abstains instead of hallucinating. LangGraph + hybrid search + reranking, served over MCP, measured against two naive-RAG baselines with an LLM-as-judge harness and RAGAS.
MIT -
Text2SQL data-analysis agent: three handwritten LangGraph orchestrations (deterministic pipeline / autonomous ReAct loop / Supervisor multi-agent with LLM-as-Judge), 4-layer SQL safety, MCP server+client, human-in-the-loop write path with transactional dry-run. 60-case gold benchmark.
MIT -
BestTestergithub.com/nshportun/BestTesterupdated this year
Production-grade Playwright + TypeScript QA framework with AI-powered testing, LLM-as-Judge evaluation, MCP server, 7 CLI agents, security fuzzing, CI/CD pipelines, Jira sync, and Slack reporting — zero-config, plug-and-play.
Apache-2.0 ★ 11 -
code-review-crewgithub.com/siddlpskmi/code-review-crewupdated this year
Autonomous multi-agent AI code reviewer with custom MCP server, agentic RAG, and LLM-as-judge evals
-
frontkitgithub.com/lakshmi930/frontkitupdated this year
MCP server + eval gate for four small AI tools. Deterministic checks and an LLM judge fail the build when model behaviour drifts.
-
research-pilotgithub.com/Marwaguerfel/research-pilotupdated this year
Autonomous AI research agent: parallel planning, RAG memory, MCP server, LLM-as-judge eval, Streamlit UI
-
annuma-study-companiongithub.com/AlirezaFakari/annuma-study-companionupdated this year
AI active-recall study agent for AnNuMa — MCP server, read-only guardrail, LLM-as-judge
★ 1 -
Prompt-injection guardrail with a hardened LLM judge, guarded RAG and an MCP server · FastAPI + React/TS
MIT -
RAG support agent for banking security FAQs — LangGraph, ChromaDB, MCP, LLM-judged faithfulness eval in CI (16/16, p95 1.9s)
-
MailTunegithub.com/bazzal99/MailTuneupdated this year
QLoRA fine-tuned email assistant with a full evaluation suite: pairwise LLM-judge comparison, fact-preservation testing, generalization checks, and FastAPI/MCP serving.
★ 1 -
cordis-mcp-servergithub.com/tavitatavi/cordis-mcp-serverupdated this year
FastMCP server exposing read-only, typed tools over a DuckDB warehouse of CORDIS EU research-funding data, plus a LangGraph agent and an LLM-judged evaluation harness.
★ 1 -
RAG over SEC 10-K/10-Q filings evaluated on FinanceBench: retrieval ablation, cited answers, LLM judge, citation check, MCP server
MIT -
agents-AIgithub.com/RenanMiqueloti/agents-AIupdated this year
Agentes LangGraph com HITL via interrupt(), MCP server custom e evals LLM-as-judge — multi-provider (Claude, OpenAI, Ollama)
MIT -
k11techlab-llm-assurance-toolkitgithub.com/K11-Software-Solutions/k11techlab-llm-assurance-toolkitactive
LLM red-teaming & assurance toolkit: 41 OWASP LLM Top 10 probes, CI quality gate, audit-ready HTML reports, Claude LLM-judge and an MCP server for agent-driven red-teaming.
pypi MIT ★ 2 -
Multi-tenant B2B call analytics platform — LangGraph pipeline, MCP server, pgvector RAG, JWT auth with Postgres Row-Level Security, and automated LLM-as-judge eval harness
-
An LLM eval harness graded by computed ground truth, not an LLM judge. Worked example: poker, served to models as MCP tools — gpt-4o-mini goes from 26/75 to 75/75 with them.
MIT ★ 1 -
Don't trust AI. Verify it. Open-source verification for evidence-grounded AI — deterministic grounding checks against source evidence, no LLM judge. CLI, GitHub Action, Python, MCP server, Docker.
go MIT ★ 1 -
rag-eval-enginegithub.com/aptsalt/rag-eval-engineupdated this year
Production RAG with continuous evaluation — hybrid retrieval (BM25 + vector/RRF), LLM-as-judge scoring on faithfulness/relevance/hallucination, MCP server, adaptive retrieval tuning. 68 tests.
MIT -
Local-first assistant that watches your Jira/Slack/GitHub and learns which tools actually solve which problems — an agentic loop graded mechanically, no LLM-as-judge.
NOASSERTION -
An MCP server that checks an AI agent's draft answer against its sources: figures, dates, citations and must/may wording, sentence by sentence; an optional LLM judge reads the prose.
pypi AGPL-3.0-only -
nocapgithub.com/xi029/nocapactive
No evidence, no answer. 🧢 System One evidence gate for RAG & AI agents: Jev / Laya / any LLM judges before your LLM speaks. MCP server, Python SDK, CI route tests.
Apache-2.0 ★ 1 -
Witness — browser acceptance checks for agent-built web apps. CLI, MCP, GitHub Action. The browser is the judge, not an LLM.
npm MIT ★ 2 -
Structural framing analysis as an MCP server with deterministic detection. The caller's agent runs the judge, so Frame Check bears zero LLM cost per query.
pypi Apache-2.0 -
Measured RAG and agent over Swiss bank annual reports: retrieval lab, validated LLM judge, guarded text-to-SQL, MCP tools behind a deterministic permission layer, prompt-injection suites, self-hosted model comparison.
NOASSERTION -
Evidence-grounded IT support agent: Gemini function calling over an MCP-style tool server (KB search, ticket search, ticket drafts), with a 60-question eval harness, LLM judge, and OpenTelemetry tracing.
★ 1 -
A from-scratch Model Context Protocol server (stdio, stdlib-only, zero deps): safe calc (OWASP LLM06), BM25 search, a no-LLM-judge answer grader, and live model-drift / eval-run lookups. 42 tests.
MIT ★ 1 -
MCP server that checks an AI agent's answer against its tool outputs before you see it: exact checks, NLI and an LLM judge, with block / annotate / rewrite policies
MIT -
Agentic-Document-Intelligence-Industrialized-Cross-Reference-Enginegithub.com/gayathrikumari/Agentic-Document-Intelligence-Industrialized-Cross-Reference-Engineupdated this year
🤖 Built a production-grade Agentic AI system using MCP + ReAct Multi-Agent Patterns to automate complex engineering document cross-referencing | Multimodal RAG | LLM-as-a-Judge | KV Caching
-
Ticket tracker for AI and human work: Go + GraphQL + Postgres, SolidJS UI, and an MCP server so Claude Code agents can claim, file and work tickets, with LLM-as-judge evals.
MIT -
RAG-System-from-Scratch-avec-MCPgithub.com/FEDI-HASSINE/RAG-System-from-Scratch-avec-MCPupdated this year
Système RAG complet construit de zéro avec recherche sémantique, reranking, agent IA, API MCP, évaluation LLM-as-a-Judge et interfaces CLI/Streamlit.
★ 1 -
Ask questions about your PDFs and get answers with page-level citations. Hybrid FAISS + BM25 retrieval with Reciprocal Rank Fusion, LLM-judged evaluation (18/20 correctness, 20/20 faithfulness) and an MCP server for AI clients.
MIT ★ 1 -
Open-source, self-hosted observability for AI agents: OTLP ingestion, tracing SDK, sessions, cost analytics, datasets and LLM-as-judge experiments, alerts, replay, and an MCP server. SQLite or PostgreSQL.
pypi MIT -
tardygradagithub.com/fabio-rovai/tardygradaupdated this year
A verified runtime for trustworthy AI agents: every claim passes an 8-layer verification pipeline before it counts. Deterministic checks, explainable verdicts, no LLM judging an LLM. Pure C, zero dependencies, MCP-native.
MIT ★ 26 -
cdvgithub.com/azank1/cdvupdated this year
Conservative Dual-Verify — judging AI agent work with a deterministic floor + LLM critic (stricter score wins) and Bayesian stopping. Research preview, frozen at v1.0.1; active development continues privately. MIT.
pypi MIT ★ 4 -
ripostegithub.com/RishiSastry/riposteupdated this year
A DSL for Pokémon battle policies (Rust compiler, Python runtime, MCP server), built to measure which steering strategy best teaches a coding agent a language it has never seen. Graded by real battle wins, not an LLM judge.
-
A production-style multi-agent system that generates context-aware briefs. Built on MCP + A2A, with persistent memory (Chroma), dynamic tool routing, self-critique, distributed tracing (Arize Phoenix), an LLM-as-judge eval harness, and Redis caching.
★ 1 -
Self-hosted AI web mail for all your mailboxes: an LLM judges what matters across inboxes and spam, pushes only that, and the web view opens on what needs you, to read, file and answer there. MCP server included.
MIT -
fieldraggithub.com/woongjaejung/fieldragupdated this year
Grounded RAG + agent assistant for NGS field support on Google Cloud (Gemini + pgvector/Cloud SQL + FastAPI/Cloud Run, LangGraph, MCP, LLM-judge eval)
-
A production-grade, self-hostable MCP server for Quera, bringing Quera’s problemset, submissions, courses, contests, LMS, AI, judge tooling, and authoring workflows to LLM agents. Built with async Python, 259+ tools, RBAC, safety gates, audit logs, revisions, multi-account support, and Docker deployment.
MIT -
Three-agent research system on the OpenAI Agents SDK — Planner/Executor/Reviewer with typed handoffs, a custom MCP server, function tools, input/output guardrails, token-streaming (CLI + SSE), tracing, and an LLM-judge eval harness with a pytest regression gate.
-
DODI: a deterministic Terms of Service deception scorer with an MCP server. A 2015-2024 study of 10 platforms with ToS;DR and LLM-judge validation, plus v2, which weighs each contract against what the store promised.