MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. Each candidate is checked by a language model from its name, description and topics, and only actual MCP servers are listed (not clients, SDKs, lists or skills). Package descriptions, licences and links are read from each package's own registry (npm, PyPI, crates.io, NuGet, RubyGems; Go and Maven packages are only checked to exist), and packages the registry no longer has are removed. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
llm-as-judge-mcp-servergithub.com/jgsheppa/llm-as-judge-mcp-serverupdated this year
An MCP server to judge an LLM's responses
npm MIT -
mcp-as-a-judgegithub.com/OtherVibes/mcp-as-a-judgeupdated this year
MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations
pypi MIT ★ 17 -
LLM-as-Judge 텍스트 품질 평가를 MCP 서버로 — 루브릭 채점·골든셋 시험·교차 검증
MIT ★ 1 -
Judges agent/RAG outputs (relevance, hallucination, accuracy, routing, prompt injection) using Laya instead of LLM-as-judge -- usable as an MCP server or as a plain Python library.
pypi MIT -
Governance-classifier eval harness on the Claude Messages API, with calibrated LLM-as-judge and an MCP server
MIT -
mcp-judgegithub.com/jddavenportOpen/mcp-judgeupdated this year
A calibrated multi-persona LLM-as-judge, exposed as an MCP server and shipped with an eval harness that proves the calibration.
MIT -
企业级智能客服 AI 系统参考实现:三路意图融合 · MCP 工具调用 · 三级记忆 · LangGraph 多 Agent 编排 · Monitor 闭环 · LLM-as-Judge 评测。零密钥可运行,含两引擎对照数据与诚实记录。
MIT ★ 3 -
JevLab: Offline LLM-as-Judge evaluation, position/verbosity bias diagnostics, Platt calibration & agreement benchmark studio (MCP + CLI + UI)
Apache-2.0 ★ 1 -
MCP server for automated insurance claim processing — LLM-as-judge pattern, Agent Skills, TypeScript
★ 1 -
Persistent memory + workflow orchestration as an MCP server. 78 canonical tools across 21 namespaces, SQLite-backed, with an LLM-as-judge drift gate and a vibe-flow loop (spec to artifact to drift to publish).
npm NOASSERTION ★ 2 -
A RAG agent that grades its own retrieval, reformulates the query when confidence is low, and abstains instead of hallucinating. LangGraph + hybrid search + reranking, served over MCP, measured against two naive-RAG baselines with an LLM-as-judge harness and RAGAS.
MIT -
Human-in-the-loop LLM Failure Detection & Benchmark Toolkit — Auto-flag LLM failures (hallucination, refusal, factual error) with LLM-as-Judge explanations. Confirm them in a review UI or via MCP agent — then export a citable benchmark validated against expert labels. Compatible with Ollama and Amazon Bedrock models.
-
Text2SQL data-analysis agent: three handwritten LangGraph orchestrations (deterministic pipeline / autonomous ReAct loop / Supervisor multi-agent with LLM-as-Judge), 4-layer SQL safety, MCP server+client, human-in-the-loop write path with transactional dry-run. 60-case gold benchmark.
MIT -
BestTestergithub.com/nshportun/BestTesterupdated this year
Production-grade Playwright + TypeScript QA framework with AI-powered testing, LLM-as-Judge evaluation, MCP server, 7 CLI agents, security fuzzing, CI/CD pipelines, Jira sync, and Slack reporting — zero-config, plug-and-play.
Apache-2.0 ★ 11 -
code-review-crewgithub.com/siddlpskmi/code-review-crewupdated this year
Autonomous multi-agent AI code reviewer with custom MCP server, agentic RAG, and LLM-as-judge evals
-
research-pilotgithub.com/Marwaguerfel/research-pilotupdated this year
Autonomous AI research agent: parallel planning, RAG memory, MCP server, LLM-as-judge eval, Streamlit UI
-
annuma-study-companiongithub.com/AlirezaFakari/annuma-study-companionupdated this year
AI active-recall study agent for AnNuMa — MCP server, read-only guardrail, LLM-as-judge
★ 1 -
agents-AIgithub.com/RenanMiqueloti/agents-AIupdated this year
Agentes LangGraph com HITL via interrupt(), MCP server custom e evals LLM-as-judge — multi-provider (Claude, OpenAI, Ollama)
MIT -
Multi-tenant B2B call analytics platform — LangGraph pipeline, MCP server, pgvector RAG, JWT auth with Postgres Row-Level Security, and automated LLM-as-judge eval harness
-
An LLM eval harness graded by computed ground truth, not an LLM judge. Worked example: poker, served to models as MCP tools — gpt-4o-mini goes from 26/75 to 75/75 with them.
MIT ★ 1 -
rag-eval-enginegithub.com/aptsalt/rag-eval-engineupdated this year
Production RAG with continuous evaluation — hybrid retrieval (BM25 + vector/RRF), LLM-as-judge scoring on faithfulness/relevance/hallucination, MCP server, adaptive retrieval tuning. 68 tests.
MIT -
Local-first assistant that watches your Jira/Slack/GitHub and learns which tools actually solve which problems — an agentic loop graded mechanically, no LLM-as-judge.
NOASSERTION -
RAG-System-from-Scratch-avec-MCPgithub.com/FEDI-HASSINE/RAG-System-from-Scratch-avec-MCPupdated this year
Système RAG complet construit de zéro avec recherche sémantique, reranking, agent IA, API MCP, évaluation LLM-as-a-Judge et interfaces CLI/Streamlit.
★ 1 -
Structural framing analysis as an MCP server with deterministic detection. The caller's agent runs the judge, so Frame Check bears zero LLM cost per query.
pypi Apache-2.0 -
Agentic-Document-Intelligence-Industrialized-Cross-Reference-Enginegithub.com/gayathrikumari/Agentic-Document-Intelligence-Industrialized-Cross-Reference-Engineupdated this year
🤖 Built a production-grade Agentic AI system using MCP + ReAct Multi-Agent Patterns to automate complex engineering document cross-referencing | Multimodal RAG | LLM-as-a-Judge | KV Caching
-
Ticket tracker for AI and human work: Go + GraphQL + Postgres, SolidJS UI, and an MCP server so Claude Code agents can claim, file and work tickets, with LLM-as-judge evals.
MIT -
Open-source, self-hosted observability for AI agents: OTLP ingestion, tracing SDK, sessions, cost analytics, datasets and LLM-as-judge experiments, alerts, replay, and an MCP server. SQLite or PostgreSQL.
pypi MIT -
A production-style multi-agent system that generates context-aware briefs. Built on MCP + A2A, with persistent memory (Chroma), dynamic tool routing, self-critique, distributed tracing (Arize Phoenix), an LLM-as-judge eval harness, and Redis caching.
★ 1 -
judge-audit-mcpgithub.com/asif786ka/judge-audit-mcpupdated this year
Judge QA for agents — measures whether your LLM judge agrees with humans, drifted between runs, or is biased, before you ship the number
pypi MIT -
Typed judgment tools for MCP agents. TypeSafe's Jev model as verify, screen, find, classify, rerank, decide, compare, extract, review, gate, and score: the model judges, policy decides auto, review, or escalate.
MIT ★ 49 -
Guardrails for LLM answers. GroundLens checks an LLM answer against its sources without a model, decides PASS, REVIEW or FAIL under your policy, and keeps a signed record.
Apache-2.0 -
Guardrails for LLM answers. GroundLens checks an LLM answer against its sources without a model, decides PASS, REVIEW or FAIL under your policy, and keeps a signed record.
pypi Apache-2.0 ★ 8 -
The LLM Council works together to answer your hardest questions
pypi MIT ★ 49 -
Typed decisions for AI agents as an MCP tool: yes/no probability (noul), choice, and score in one fast request. Backed by the TypeSafe System One model.
npm Apache-2.0 ★ 5 -
A blind, sealed LLM-judge panel: the author never grades their own work, and every way to re-roll a grade is refused by code.
Apache-2.0 -
Pluralistic human evaluation infrastructure for AI in production
pypi MIT ★ 2 -
Benchmark LLMs on your own scenarios with a multi-judge panel. Full MCP server (tools, resources, prompts, progress) plus local GUI, CLI, and desktop app. One OpenRouter key.
MIT -
Ask several LLMs the same question, have them grade each other blind, and measure how biased the graders are. A provider-agnostic LLM Council with judge-bias analytics, CLI, MCP server and web UI.
MIT ★ 1 -
atlasgithub.com/2005-Aneeshdutt/atlasupdated this year
Automated LLM red-teaming pipeline: a LangGraph agent runs jailbreak, prompt-injection, hallucination and bias attacks against a model, scores responses with an LLM judge, mutates failed attacks, and reports results in a Streamlit dashboard and MCP agent.
MIT ★ 1 -
Let your coding agent fix a prompt until it actually works, and prove it did. A self-hosted MCP server: cross-vendor judges, frozen baselines and memory.
Apache-2.0 ★ 10 -
Is your LLM judge a coin flip? One command tests it for position bias, flip-flopping and padding bias, no labels needed, then turns your real logs into an eval suite.
MIT -
llmswapgithub.com/sreenathmmenon/llmswapupdated this year
Ask, compare, and synthesize across 10 LLM providers. Python SDK, CLI, local Arena, MCP, and privacy-aware
pypi MIT ★ 36 -
Runtime LLM verifier — family-different, reasoning-stripped, multi-lens adjudication with signed Ed25519 receipts (CLI · MCP · HTTP).
npm MIT -
DeeWaanAI — local-first MCP server and visual pipeline designer. Runs entirely on your machine; your keys, your data, your models.
npm AGPL-3.0 -
The plug-and-play evals layer for LLM-based projects: library, learning center, check center, API, MCP server.
MIT ★ 1 -
poc_scrapergithub.com/pjnhek/poc_scraperupdated this year
Grounded GTM account research: ICP scoring and citation-traced outreach, exposed as a live hosted MCP server. Every claim traces to a numbered retrieval or gets dropped.
MIT -
errorbar MCP server — every platform API (evals, criteria, gates, logs, datasets, aliases, audit, proving) as tools for Claude, Cursor, and any MCP client.
npm Apache-2.0 -
정부 R&D 매뉴얼 5권을 인쇄 쪽번호 인용과 함께 답하는 RAG. 하이브리드 검색, 6방식 벤치마크, 로컬 MCP로 Claude Desktop 배포
-
Self-hosted multi-agent AI platform: generates, evaluates and refines LLM answers until they hit a quality bar (~22% more accurate, ~45% fewer hallucinations)
MIT -
quorumgithub.com/danielhansenjones/quorumupdated this year
Multi-agent RAG system on LangGraph: parallel per-axis analyst agents over hybrid retrieval (Qdrant, BGE-M3) and Postgres, a tool-using critic for claim-level fact-checking, MCP server, and an eval harness with A/B campaigns and a QLoRA-distilled local judge.
MIT -
The agent that turns a pile of context into a defensible comparison: one MCP tool surface, two runtimes, span-level citations and honest uncertainty. Produces comparanda analyses.
Apache-2.0 -
Test each claim in a thesis against a measurement and get MATCH, DRIFT or UNVERIFIABLE, with a sealed record anyone can recheck offline. No model in the verdict step.
pypi NOASSERTION ★ 3 -
Agente especialista que gera, executa e monitora algoritmos sob demanda, com análise de complexidade, em uma interface de chat.
-
A Swiss Army knife that lets AI agents get the most out of decision models, simply. Typed yes/no, choice and score judgments over MCP, REST and a CLI, to gate and triage agent work. Written in Rust.
MIT ★ 2 -
MCP server that gives AI agents typed decisions instead of prose: classify, route, score, and flag with calibrated probabilities. npx -y openjev-mcp
npm MIT ★ 1 -
@ezark-publish/agentdesk-mcpgithub.com/rih0z/agentdesk-mcpupdated this year
MCP server for AgentDesk AI-to-AI Service Marketplace. Quality review, service catalog, and marketplace execution — all via MCP tools.
npm MIT -
MCP server and Claude Code plugin for Agent Referee: independent sealed reviews of an agent’s work (free daily allowance), tie-breaks between agents, and offline checks of signed receipts.
npm MIT ★ 1 -
Jev MCP server: a decision layer for coding agents, built on TypeSafe Jev (System One model). Ship gates, risk checks, file triage that keeps files out of context, and a safe headless browser, with calibrated confidence. For Claude Code, Codex, Cursor.
npm MIT ★ 6 -
Provider-neutral prompt calibration engine with sensitivity-ranked meta-axes, hard-gate x soft-score fitness, walk-forward ship gates, guarded/expedition profiles, and explicit capability/risk reporting across Anthropic, OpenAI, Gemini, and local OpenAI-compatible backends. Optional sub-tools (mini-omega-lock, mini-antemortem-cli) distribute separately and plug in via the omegaprompt.preflight interface.
pypi Apache Software License -
fusion-mcpgithub.com/yksanjo/fusion-mcpupdated this year
Zero-dep MCP server exposing OpenRouter Fusion (multi-model panel + judge synthesis) as a tool any agent can call.
MIT