MCP servers that ship as code — a repository or a package you install and run locally (usually over stdio). They have no hosted endpoint, so we cannot probe or grade them: instead each one shows its repository, install command, license, stars and when it was last updated.
Collected from GitHub repositories and npm, PyPI, Cargo, Go, Maven, NuGet and RubyGems packages tagged as MCP servers, the official MCP Registry, Docker's MCP catalog, ModelScope, Baidu MCP World, Cline and other public directories; one entry per repository. A server that also offers a hosted endpoint is listed under Remote instead. These entries are not counted in our reachability, verification or readiness statistics.
-
llm-benchmark-mcp-servergithub.com/AiAgentKarl/llm-benchmark-mcp-serverupdated this year
MCP Server for LLM comparison, benchmarks, and pricing — find the best model for any task
pypi MIT ★ 1 -
Detect pytest config-leakage that corrupts agent-benchmark grading. Deterministic, no-LLM. Catches the VulcanBench --cov addopts mis-scoring bug.
pypi MIT -
Local-first MCP multi-LLM orchestrator with task decomposition and benchmark-guided model routing through OpenRouter.
MIT ★ 5 -
metrillmgithub.com/MetriLLM/metrillmupdated this year
Benchmark local LLM models: speed, quality, and hardware fitness scoring. CLI, MCP server, and IDE plugins.
npm Apache-2.0 ★ 8 -
Benchmark AI model inference speed — like speedtest.net but for LLMs. MCP server with zero telemetry.
MIT -
Benchmark that scores LLMs on fixing Arma 3 SQF bugs through three MCP tools; the traps and the results table stay server-side.
MIT -
MCP server for the PipelineScore LLM benchmark — exposes run_benchmark / get_user_leaderboard / get_user_profile tools to any MCP-compatible client (Claude Code, Codex, Cursor, Continue, Cline). Lets your AI drive local-first LLM benchmarking on your hard
npm Apache-2.0 -
topgameai-mcp-servergithub.com/hezeclark/ai-arenaupdated this year
MCP server for AI Arena — benchmark your LLM across 9 classic games
npm MIT -
Availability, latency, load, capability and quality benchmarks for any OpenAI-compatible LLM API — interactive dashboard, detailed reports, MCP server and installable AI skill
AGPL-3.0 ★ 1 -
MCP server for measured local-LLM benchmarks from the AGmind claim registry — AMD Strix Halo (Ryzen AI Max+ 395) llama.cpp numbers today, NVIDIA DGX Spark on the bench. search_claims, get_claim, list_measured. Read-only, CC BY 4.0 data.
Apache-2.0 ★ 4 -
JevLab: Offline LLM-as-Judge evaluation, position/verbosity bias diagnostics, Platt calibration & agreement benchmark studio (MCP + CLI + UI)
Apache-2.0 ★ 1 -
Index a video once, query it cheaply with any LLM. Open model-free video context format (.vctx) + MCP server + accuracy-per-dollar benchmark.
pypi Apache-2.0 ★ 1 -
Human-in-the-loop LLM Failure Detection & Benchmark Toolkit — Auto-flag LLM failures (hallucination, refusal, factual error) with LLM-as-Judge explanations. Confirm them in a review UI or via MCP agent — then export a citable benchmark validated against expert labels. Compatible with Ollama and Amazon Bedrock models.
-
Streaming privacy for LLM apps: pii-leak-benchmark (does your gateway leak PII in streams?), chunk-invariance (one-line test for split-boundary leaks), mcp-ssrf-check, and a self-hosted redacting gateway.
pypi Apache-2.0 ★ 17 -
LLM-friendly API cards, an MCP server and a GPU benchmark for CuPy and nvmath-python (cuBLAS, cuFFT, cuSOLVER, cuSPARSE, cuDSS, cuTENSOR)
MIT -
Pluggable, observable RAG MCP server — hybrid retrieval (BM25+dense+RRF), BEIR-benchmarked rerankers (BGE / TypeSafe Jev / LLM), Streamlit dashboard + Langfuse, multimodal ingestion
-
Official SDKs for the AI Stupid Level Smart Router (benchmark-driven LLM router) and Data API: TypeScript, Python, Vercel AI SDK provider, MCP server, LangChain, n8n.
npm MIT -
Benchmark LLMs on your own scenarios with a multi-judge panel. Full MCP server (tools, resources, prompts, progress) plus local GUI, CLI, and desktop app. One OpenRouter key.
MIT -
Serve, size and benchmark open LLMs on one 16 GB GPU: vLLM vs llama.cpp vs Ollama, KV cache and offload planning, MCP server
MIT -
ToolRoutegithub.com/grossiweb/ToolRouteupdated this year
Intelligent routing layer for AI agents: recommends the best MCP server and LLM for any task, scored on 132+ real benchmark executions.
npm MIT ★ 2 -
Benchmark local LLM inference speed (tokens/sec) on your own hardware — llama.cpp native + cloud APIs, 124-model catalog, optimal-quant picker, and an MCP serve mode.
MIT ★ 2 -
Benchmark and MCP tools for LLM agents doing root-cause analysis on OpenTelemetry metrics, traces and logs. 20 recorded microservice incidents, every run scored against a 28-line rule and replayable. Only the strongest models clearly win.
Apache-2.0 -
AI-powered climate intelligence system using MCP tools, RAG pipelines, and LLMs to answer climate risk queries. Benchmarked on 20 gold cases with 70% accuracy across BWS, RFR, and DRR indicators.
-
hpml-assetopsbench-smart-grid-mcpgithub.com/eggrollofchaos/hpml-assetopsbench-smart-grid-mcpupdated this year
SmartGridBench: a smart-grid transformer-maintenance benchmark for tool-using LLM agents, built on IBM AssetOpsBench with MCP/direct-call tool surfaces.
MIT -
hpml-assetopsbench-smart-grid-mcpgithub.com/HPML6998-S26-Team13/hpml-assetopsbench-smart-grid-mcpupdated this year
SmartGridBench: a smart-grid transformer-maintenance benchmark for tool-using LLM agents, built on IBM AssetOpsBench with MCP/direct-call tool surfaces.
MIT ★ 1 -
Measure what context compression actually costs an LLM agent: where a session's tokens go, and whether a rewrite pays for the prompt cache it invalidates. Rust crate, MCP server and benchmark harness.
npm MIT ★ 1 -
GRPO-based multi-objective analog circuit sizing on AnalogGym benchmarks (ngspice + SkyWater sky130), with LLM-agent integration via Claude Code skills and an MCP server. Companion code for the IEEE TCAD paper AnalogGym-Opt (doi 10.1109/TCAD.2026.3733612).
MIT ★ 6 -
Safe I/O harness for LLM agents: serial, Modbus TCP, MQTT, file sandbox and ESP32 flashing behind an MCP server, with a JSONL audit journal - plus ironbench, a pass@k benchmark for firmware agents on Wokwi/Renode/real hardware.
MIT ★ 2 -
Benchmark the full voice-AI pipeline on your own audio — STT (batch + streaming), LLM TTFT, speech-to-speech, and TTS. WER by failure mode, keyterm recall, real-time-paced latency, cost columns, a CI regression gate, and an MCP server for coding agents.
MIT ★ 1 -
Can LLMs learn a programming language they've never seen? A from-scratch compiler for MiniLang, used as an uncontaminated benchmark where Claude, GPT and Qwen learn it from the spec and debug through a compiler-as-tool agent loop. MCP server included.
-
Auditable memory layer for AI agents: zero-LLM-call local ingest (~10ms/msg, air-gapped), matches Mem0 on accuracy at ~1000x lower ingest cost, bi-temporal belief-state, MCP server. Honest LoCoMo/LongMemEval benchmarks. Open source (Apache-2.0).
npm Apache-2.0 ★ 12 -
Long-term memory for LLM assistants: LLM extraction with JSON-schema validation, hybrid recall of memories (dense + BM25, RRF) plus raw turns, served over HTTP and MCP, and a LoRA-distilled Qwen3.5-0.8B extractor that runs locally via Ollama. Benchmarked on LoCoMo. Rebuilt from a 26th-place Neurohack (IIT Guwahati) entry.
★ 1 -
crucible-benchgithub.com/romariokavin1/crucibleupdated this year
Run an AI trading agent (any LLM provider) against Crucible Bench scenarios on 0G — single-command MCP+EIP-712 benchmark CLI.
npm MIT -
constellategithub.com/rjdscott/constellateupdated this year
Three knowledge planes (embedded, unified, composed) behind one retrieval contract, benchmarked head-to-head on MovieLens ml-25m. Relational, vector, and graph fusion, an MCP context plane with dual LLM drivers, and a React explorer.
MIT ★ 1 -
MCP server for local LLM inference on Ryzen AI Max+ 395 with queueing, deadlines and telemetry; benchmark findings
MIT -
A2A v1.0 multi-vendor agent exchange: JWS-signed Agent Cards with pinned vendor keys, deterministic trust and quarantine, cost- and reliability-aware broker over A2A, MCP tools, metered local LLMs, adversarial benchmark reported as n/N.
Apache-2.0 -
Will this LLM run on your machine, and how fast? One command: memory check, speed estimate, real benchmark, local UI, MCP server.
MIT -
mcp-dungeongithub.com/appflavor/mcp-dungeonupdated this year
Text-based dungeon crawler playable by LLM agents via MCP — with a live spectator page and benchmark league
MIT -
LLM cost calculator, token counter, latency benchmark, CI guardrail, MCP server, and VS Code/Cursor extension.
npm MIT ★ 1 -
Autonomous Prompt Engineering, Jailbreak Defense, LLM Cost Optimizer & Eval Benchmark MCP for Cursor, Claude & Windsurf
-
voiceforge-mcpgithub.com/beknazar/voiceforge-mcpupdated this year
MCP server for voice AI development — recommend, benchmark, and scaffold STT+LLM+TTS stacks
npm -
LLM agent that turns natural-language requests into sandboxed Monte Carlo simulation code, with self-correction, MCP server and benchmark
MIT -
Long-term memory for LLM agents: hybrid BM25 + vector retrieval that learns from use, a measured retrieval benchmark, and a from-scratch MCP server. Standard library only.
MIT -
Persistent memory for LLM agents — episodic/semantic/procedural memory, auditable forgetting, and a benchmark proving it actually recalls things days later.
MIT -
Local AI support investigator with RAG, MCP tools, deterministic evidence verification, and benchmarked local LLM workflows.
-
voting-mcpgithub.com/HrishiKabra/voting-mcpupdated this year
MCP server exposing principled social-choice aggregation rules + a benchmark measuring their accuracy vs majority vote over an LLM ensemble
pypi -
LLM agent that triages GitHub issues (type labels, duplicates) through its own MCP server, benchmarked on 120 real astral-sh/uv issues: zero-shot vs RAG vs agent.
MIT -
Correctness-gated, LLM-driven GPU kernel optimization on the Raspberry Pi 5's V3D GPU. The model proposes; a Burr state machine served over MCP verifies against physics gates, benchmarks, and decides.
MIT -
Agentic ADaM code generation with QC-grade verification: LangGraph, MCP, multi-model LLMs, CDISC, human sign-off and a public pharmaverse benchmark
-
MCP server for delegating tasks across LLM backends and benchmarking their outputs
npm MIT -
Token-saving browser automation MCP server: cached actions, then a local LLM, then Claude. Ground-truth benchmark included; 0 Claude tokens across 20 real runs.
-
llm-guard-gatewaygithub.com/ShreyAC962/llm-guard-gatewayupdated this year
Governance gateway for LLM and agent traffic: blocks prompt injection, redacts PII, enforces per-tenant token budgets, and serves repeat prompts from a semantic cache, cutting served p50 from 44.2 ms to 4.6 ms in the included benchmark. REST + MCP surfaces, offline-first local profile.
MIT -
Convert PDF, Word, Excel, PowerPoint and HTML to Markdown for LLMs and RAG. Rust CLI, Python API and MCP server, with published compatibility tests and benchmarks.
cargo MIT ★ 4 -
Midasgithub.com/vornicx/Midasupdated this year
Local-first, eval-first memory for long-horizon AI agents — no LLM at ingest. Python SDK + MCP server with source-traceable recall, belief revision, selective forgetting, and reproducible benchmarks.
pypi MIT ★ 14 -
zabbix-mcp-lite: a Zabbix 8.0 MCP server built for small local LLMs. With it, 4B models answered CPU/memory history questions 16/16, vs 0/16 with the official MCP tools. Docker stack with LibreChat + LM Studio and a reproducible benchmark.
MIT ★ 1 -
Evidence-gated execution for AI agents: a task succeeds only when the host verifies it. Hash-chained logs, a fault-injection benchmark, and a hidden-test study of LLM 'done' claims.
MIT -
Text2SQL data-analysis agent: three handwritten LangGraph orchestrations (deterministic pipeline / autonomous ReAct loop / Supervisor multi-agent with LLM-as-Judge), 4-layer SQL safety, MCP server+client, human-in-the-loop write path with transactional dry-run. 60-case gold benchmark.
MIT -
aa-mcpgithub.com/Leev1s/aa-mcpupdated this year
MCP server wrapping the Artificial Analysis public API. Enables AI agents to query LLM and multimodal model benchmarks, pricing, speed data, and track model updates via structured diffs.
pypi MIT ★ 3 -
perfdigest-mcpgithub.com/onlyxitachi/perfdigest-mcpupdated this year
Local MCP server that digests development-loop reports — profilers (NVIDIA ncu, AMD rocprof, Linux perf, Apple Metal, chrome-trace), build tools (ptxas, clang -ftime-trace, cmake, ninja, cargo), benchmarks (criterion), CI run logs, and git change state — into token-efficient JSON for LLM coding agents.
pypi Apache-2.0 -
prestyj-stats-mcpgithub.com/Gahroot/prestyj-stats-mcpupdated this year
MCP server exposing the Prestyj open statistics dataset (CC BY 4.0). Plug into Claude Desktop, Cursor, Cline, Continue, Zed — your LLM gets native search + cite over 58 lead-response, video-ad, AI-sales benchmarks.
go MIT