MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
llm-chess-evaluation-harnessgithub.com/PythonicVarun/llm-chess-evaluation-harnessupdated this year
LLM Chess Evaluation Harness ♟️
MIT -
Cloudflare Workers MCP server wrapper: llm-output-evaluator
npm MIT -
MCP server for agentic LLM evaluation: jury scoring, agent tracing via OpenTelemetry, document-grounded QA generation, PDF reports.
pypi Apache-2.0 -
RepoLens: issue-to-code search with a local MCP server and reproducible SWE-bench evaluation
-
A local studio to tune and extend language models: versioned datasets, LoRA and QLoRA training on the allowed GPUs, merges, GGUF conversion, importance-matrix quantization, YaRN context extension and evaluation against the base model. MCP tools for assistants.
MIT -
A lightweight, pure web search solution for large language models, supporting multi-engine aggregated search, deep reflection and result evaluation. A balanced approach between web search and deep research, providing a framework-free implementation and mcp server for easy developer integration.
Apache-2.0 ★ 2 -
MCP server for LLM evaluation gates.
pypi MIT -
Scenario-driven framework for evaluating demographic bias in single- and multi-agent LLM workflows. Tracks bias across sequential stages, separates model non-determinism from systematic effects, and prevents agents inferring evaluation intent. REST API and MCP server.
MIT -
Self-hosted multi-agent AI platform: generates, evaluates and refines LLM answers until they hit a quality bar (~22% more accurate, ~45% fewer hallucinations)
MIT -
Open source AI evaluation framework — LLM-as-judge + assertion-based evals for any AI app. CLI + MCP server.
npm Apache-2.0 -
mcp-eval-servergithub.com/Hamza2497/mcp-eval-serverupdated this year
MCP server for structured LLM output evaluation — factual accuracy, logical consistency, rubric scoring
-
MCP server for Bench — evaluate and optimize your LLM prompts from any MCP client
npm MIT -
Stateless MCP 2026-07-28 server + evaluation of LLM tool selection at scale (N=5..100, embedding retrieval) and agent loop guards on a local LLM
MIT -
MCP server for inspectable semiconductor company context, tool-based QA, and baseline-versus-LLM evaluation.
-
MCP server for Arize Phoenix — LLM tracing, evaluation, and dataset management via AI agents
MIT -
Evaluation and runtime governance layer for local LLMs (Ollama + MLX-LM): deterministic policy/provenance guardrails, MCP server, regression gate, and reviewer console
★ 1 -
Give your AI agents an evaluation tool - provider-agnostic LLM eval runner with an MCP server for Claude, structured output validation, datasets, evaluators and honest cost metrics
pypi MIT -
cordis-mcp-servergithub.com/tavitatavi/cordis-mcp-serverupdated this year
FastMCP server exposing read-only, typed tools over a DuckDB warehouse of CORDIS EU research-funding data, plus a LangGraph agent and an LLM-judged evaluation harness.
★ 1 -
Question answering over your documents with cited sources: FastAPI + SSE, retrieval over MCP (cMCP + cRAG), local LLM via llama.cpp, evaluation gated in CI
MIT ★ 1 -
JevLab: Offline LLM-as-Judge evaluation, position/verbosity bias diagnostics, Platt calibration & agreement benchmark studio (MCP + CLI + UI)
Apache-2.0 ★ 1 -
Grounded enterprise ontology agent with Neo4j, typed retrieval, LLM tool calling, evaluation, and read-only MCP tools.
-
hybrid-rag-mcp-agentgithub.com/sharmaranupama/hybrid-rag-mcp-agentupdated this year
RAG system with hybrid search, a ReAct agent, MCP tool exposure, and LLM evaluation
-
Standalone static preflight checks for LLM evaluation configurations, leakage detection and canonical risk rules.
Apache-2.0 ★ 1 -
Standalone measurement-reliability probes and empirical noise-ceiling estimation for repeated LLM evaluations.
Apache-2.0 ★ 1 -
RAG over SEC 10-K/10-Q filings evaluated on FinanceBench: retrieval ablation, cited answers, LLM judge, citation check, MCP server
MIT -
tuneforgegithub.com/JustVugg/tuneforgeupdated this year
An MCP server for dataset generation, fine-tuning, RL, and evaluation of LLMs — directly from your coding agent.
MIT ★ 17 -
Evaluation-driven LLM agent for customer operations with LangGraph, RAG, structured tool execution, policy-aware escalation, and MCP interoperability.
-
Cited-answer RAG service over 4 Czech labour laws: hybrid retrieval, LLM reranking, REST API (FastAPI) and MCP server for AI agents, offline evaluation with confidence intervals.
MIT -
binevalgithub.com/darksolitaire9-hub/binevalupdated this year
A deterministic, Rust-native binary evaluation kernel for AI safety, designed to help autonomous agents self-correct and red-team LLM infrastructure without fuzzy scores.
MIT -
mcp-as-a-judgegithub.com/OtherVibes/mcp-as-a-judgeupdated this year
MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations
pypi MIT ★ 17 -
RAG over a software project's docs and code: answers with line-level citations or an explicit refusal, measured with its own evaluation tooling. FastAPI, PostgreSQL + pgvector, local LLMs, MCP.
-
RAG + LLM maintenance copilot: LangGraph multi-agent supervisor with human approval, hybrid search over equipment manuals (Qdrant, BM25, reranking), MCP server, sensor and image analysis, evaluation suite, Langfuse tracing. Runs locally with Ollama. FastAPI, Postgres.
MIT -
rag-eval-enginegithub.com/aptsalt/rag-eval-engineupdated this year
Production RAG with continuous evaluation — hybrid retrieval (BM25 + vector/RRF), LLM-as-judge scoring on faithfulness/relevance/hallucination, MCP server, adaptive retrieval tuning. 68 tests.
MIT -
Self-hosted data annotation and labeling platform for images, video, audio, text, PDF and LLM evaluation. Your storage, your models, your cloud.
pypi NOASSERTION -
MCP server over SEC EDGAR filings with hand-written single-agent and multi-agent (planner → parallel researchers → synthesizer) LLM researchers, a live SSE UI, and a blind evaluation of when multi-agent is worth it.
-
Persistent memory and context-engineering platform for AI agents with memory intelligence, hybrid retrieval, token-budgeted context compilation, custom LLM APIs, MCP, evaluation, observability, and a React control plane.
-
Offline RAG app that explains Indian tax, bank and government notices with a local LLM. Code verifies every deadline, amount and quote against the source text. Includes an evaluation, audit log and read-only MCP server.
-
HALO (Harness for Always-On Loop Oversight), from the survey Always-On Personal Intelligence: an evaluation harness and MCP plugin for always-on personal AI assistants and LLM agents. It checks whether Sensing, Memory, and Action are connected and whether self-evolving agents learn from outcomes, via paper and code audits and ablations.
MIT ★ 8 -
Log Claude Code sessions to Opik, the open-source LLM observability and evaluation platform, built by Comet. Tracing, evaluation, and skills for observable AI applications.
go Apache-2.0 ★ 25 -
metricmate-mcpgithub.com/marilynceo/metricmate-mcpupdated this year
LLM Output Quality Evaluation u2014 quantify, evaluate, and track LLM response quality: relevance, accuracy, tone, hallucination scoring, and consistency checks
-
LLM-based HFACS classification of ASRS aviation incident reports. Includes experimental data processing, classification models, evaluation methods, and MCP server for data access.
NOASSERTION ★ 2 -
MCP server that enables LLMs to evaluate themselves
Apache-2.0 ★ 4 -
A Model Context Protocol (MCP) server exposing safe math tools to LLMs: sandboxed expression evaluation, statistics, and number theory.
go MIT -
financial-mcp-pocgithub.com/masahirosakae/financial-mcp-pocupdated this year
Business evaluation workflow PoC using MCP and local LLMs
-
Git-native AI agent co-harness — MCP server, static guards, and agentic evaluator for LLM-assisted coding
pypi MIT ★ 1 -
evalflow-mcpgithub.com/arko-14/evalflow-mcpupdated this year
MCP server for the EvalFlow API — exposes EvalFlow (LLM prompt evaluation) as tools any MCP-capable CLI or AI agent can call.
npm MIT -
chesspoint72github.com/pauliano22/chesspoint72updated this year
Agentic chess engine evaluator — MCP servers exposing perft, SPRT Elo, and tactics tools so LLM agents can autonomously validate chess engines. Point72 Cubist Hackathon 2026.
-
LLM projects: RAG with citations over mine safety regulations, an agent with a custom MCP server, LoRA/DPO fine-tuning, and an LLM evaluation harness.
-
QLoRA fine-tuned email assistant with a full evaluation suite: pairwise LLM-judge comparison, fact-preservation testing, generalization checks, and FastAPI/MCP serving.
★ 1 -
MCP server in Python that connects Claude to my AI project data: email test set and LLM evaluation results
-
RAG system with MCP server, LLM-based question answering with evaluation metrics.
AGPL-3.0 -
harnessgithub.com/p-vbordei/harnessupdated this year
Quality-gated SOP execution harness: MCP server + Claude Code plugin with separate LLM evaluator
Apache-2.0 -
ace-enginegithub.com/hassekf/ace-engineupdated this year
Architectural Coverage Engine — zero-dependency incremental architectural analysis for Laravel projects. Measures consistency, infers patterns, evaluates security baselines, and exposes an MCP server for LLM integration.
npm MIT -
MCP server giving LLM agents real authorization answers: evaluate Rego in-process via OPA, or ask an OpenID AuthZEN 1.0 PDP
go MIT ★ 1 -
AI automation portfolio project: explainable ticket triage, batch processing, audit trail, MCP 2.x, LLM integration, evaluation gates and FastAPI.
MIT -
MCP server version of the AI ROI Calculator — evaluates ROI of AI/LLM implementations using a 3-layer cost framework via Model Context Protocol
MIT ★ 1 -
Local vs. cloud LLMs triaging real Splunk alerts through the Splunk MCP Server, evaluated against manual ground truth, with a least-privilege AI identity.
-
local-dev-mcpupdated this year
Local-first MCP server that offloads code safety, boilerplate generation, dependency evaluation, project indexing, and task delegation to local LLMs via oMLX or LM Studio.
npm MIT -
MCP server for Recruitee service for easy LLM evaluation
★ 3 -
Ask questions about your PDFs and get answers with page-level citations. Hybrid FAISS + BM25 retrieval with Reciprocal Rank Fusion, LLM-judged evaluation (18/20 correctness, 20/20 faithfulness) and an MCP server for AI clients.
MIT ★ 1