Read-only BigQuery MCP server with mandatory dry-run cost caps, agent-friendly structured errors, and a Spider/BIRD-style NL-to-SQL eval harness.

—

Agent-driven eval authoring via a new authoring-loop archetype — draft → subagent-review → surgical-revise → submit, with server-enforced self-consistent graders; compiles to standard harnesses.

Claude et al. play the brilliant puzzle title "Baba is You"

An MCP server that offers 15 specialized tools to evaluate and improve the accuracy of RAG based systems.

Is your LLM judge a coin flip? One command tests it for position bias, flip-flopping and padding bias, no labels needed, then turns your real logs into an eval suite.

NIA AI Web Evaluation Agent MCP Server - Autonomous browser testing and debugging

Production deployment layer for The Gate. Multi-tenant MCP server exposing the eval engine as agent-callable tools — scored eval harness, draft→eval→revise agent example, PRD, runbook, Docker.

Eval-driven SQL reliability for AI agents — generate, validate, safely execute, and regression-test LLM SQL. MCP + FastAPI gateways, execution-truth evals, CI regression gate.

CLI and MCP server to run eval code and convert designs in the CodeConvert Figma plugin via the realtime backend

Prompt evaluation workbench as an MCP server — run evals, compare providers, and test assertions directly from Claude Code or any MCP host.

MCP server that audits how legible a web page is to AI agents, with evidence-grounded findings and evals

Jira sprint metrics computed by code, reported by an LLM, guarded against invented numbers. MCP server, evals and tracing included.

MCP server exposing portfolio tools (Semantic Search, Eval Framework, Observability) via Model Context Protocol

The local-first prompt package manager. Version control, Monaco diffing, and automated multi-model evals for AI prompts. Works 100% offline with SQLite.

Agentic reconciliation-break triage system — bounded LangGraph state machine, MCP trust-boundary tool split, human-approval interrupt, and a real eval harness.

Permission-aware MCP server and agent harness with role-scoped tools, whole-plan validation, deterministic safety evals, and a React playground.

AI-optional accounts payable agent: deterministic financial core, stateless MCP server, bilingual Next.js UI, and offline evals that run with no API key.

ragi — Retrieval that rises. Eval-first, agent-agnostic RAG toolkit: define, measure, iterate, ship as a portable retrieval contract (Python lib / MCP server) any agent consumes.

Memory and brand-context layer for AI agent systems. Multi-tenant MCP server — 39 tools auto-generated from a live OpenAPI spec. PostgreSQL, per-workspace isolation, scored eval harness.

Branch SQLite like git — copy-on-write forks, checkpoint/rollback/promote, crash-tested capture. Stock SQLite files on local disk or S3. Built for AI agents and eval harnesses.

Python framework + local Studio for reliable LLM workflows: typed files in your repo, pre-run checks, experiments and evals, MCP server for coding agents

The verification layer for the agent economy: an agent-callable primitive that checks a number before it's asserted (evals, leaderboards, backtests), returns a signed reproducibility receipt, and meters itself (prepaid credits + x402 pay-per-call).

Public Tender Intelligence Agent — unifies UK + EU procurement notices (Find a Tender, Contracts Finder, TED) into one governed dataset and serves it to AI agents via MCP, with evidence-scored evals

A transformation copilot for the first 90 days of an AI program — playbook, opportunity portfolio, costed business cases, operating model, plus the software to run it: MCP assistant with evidence-gated scoring, prototypes with eval gates, weekly autopilot (fictional cases)

LLM observability and evals in a single binary: traces, prompts, datasets, scores and review queues for LLM and agent apps. Self-hosted, SQLite inside, OpenTelemetry-native, MCP and CLI for coding agents.

Text-to-SQL agent over public Hedera mainnet data. Ask in English, get the SQL back as the citation — with an eval harness whose cases are proven to reject careless queries. Also an MCP server.

The self-improvement layer for AI agents. Ratel reads your agent's traces, finds what fails, recommends fixes and runs evals on real sessions before you ship. Self-hosted: one app, one Postgres. Includes the TypeScript and Python SDK

Read-only Kubernetes and AWS access for AI agents, behind a deny-by-default policy gate with a JSONL audit trail and eval suites that score both per-call block rate and whole-trajectory behaviour.

Research agent in Go that finds real problems people report on Hacker News, Stack Exchange and GitHub, and returns a cited report. Built from scratch on the Claude API: custom tool-use loop, context management, evals and OpenTelemetry traces.

Git-like versioning for prompts with eval-gated promotion. Store, diff, tag (draft/staging/production), A/B serve, and gate promotion on eval results. Self-hosted, not SaaS.

Offline LLM / agent eval harness with regression gates

MCP server for search over any folder of markdown, with incremental indexing and a runnable retrieval eval

MCP server in OCaml that gives AI agents live OCaml toplevels: eval, breakpoints, watches, hot-swapping, and merlin-backed navigation

Three-tool MCP server for GitHub issue triage. Shaped payloads, retries, dry-run writes, eval vs raw REST dump.

LLM planning agent over a forecasting backtest — strict tools, structured output, measured eval with failure analysis

Memory for AI agents with rules that are kept, not just written. P0-P6 layered rules enforced in code. SQLite + FTS5 + pluggable embeddings, PostgreSQL backend, MCP server, eval harness. (GitHub: p-layer)

MCP server for legal intake triage: deterministic tools, conflicts gate, eval harness

RAG over docs with verified citations, hybrid BM25 + embedding retrieval, an eval harness, and an MCP server. Python, FastAPI, Gemini/Groq.

Autonomous game playtesting agent: plays a game, hunts bugs with generic oracles + LLM, writes reproducible reports. Includes seeded-bug eval harness and MCP server.

AI-driven Universal QA Skills, MCP Server, Precision Testing (TIA), and Evals for all AI coding agents.

Give your AI agents an evaluation tool - provider-agnostic LLM eval runner with an MCP server for Claude, structured output validation, datasets, evaluators and honest cost metrics

Governed MCP + Skills agent platform over a legacy engineering-ops system: entitlements, audit, human-in-the-loop writes, eval-gated releases, OpenTelemetry tracing.

MCP server that gives merchandising agents eyes on any storefront — scrape, audit, compare, roundtable analysis, and eval tracking via 11 tools.

MCP Server: AI editors call tools over stdio/HTTP to eval JS, read the console, and simulate input on any static web page that includes client.js

The free, open watchdog for retracted & problematic citations — Retraction Watch + OpenAlex + Crossref, reconciled conservatively. MCP server + web checker, evals-backed.

Local code-intelligence MCP server for Claude Code, Codex and Cursor… Try it on your repo: uvx nexus-mcp-ci eval

Game telemetry analyst agent: plain-English questions over player data, automatic anomaly sweep (retention, funnel, economy, crashes, A/B), HTML reports. Seeded-issue eval harness + MCP server.

Open-source memory and context for user-aware agents: scoped memory, provenance, retrieval quality, correction, boundaries, evals, and MCP/HTTP access.

Evidence-grounded IT support agent: Gemini function calling over an MCP-style tool server (KB search, ticket search, ticket drafts), with a 60-question eval harness, LLM judge, and OpenTelemetry tracing.

Multi-agent AI system for hotel operations: F&B forecasting, guest memory, environmental risk. MCP-native, FastAPI, WhatsApp delivery, evals as CI merge gates.

Authenticated, self-hosted WhatsApp MCP server that serves 96 tools for the token cost of 29, with a labeled eval for tool-retrieval accuracy. Go bridge + Python FastMCP.

One shared brain for your whole AI fleet: attested memory, orchestration, governance and evals across Claude Code, Codex, Cursor, Gemini, Antigravity.

Key-free MCP servers that let LLMs build real TouchDesigner networks: local knowledge base (no API keys), offline .toe/.tox generation, live TD editing. Eval-gated - 636/636 build-token-exact.

A Jev-compatible decision layer as an MCP server: typed noul/choice/score questions with probability distributions and confidence. Pluggable backends (local needle/laya models or any OpenAI-compatible endpoint), calibration tracking, conditions v1, eval harness.

MCP server that answers questions about AI-written code from your repo and checks each file:line citation it parses. Public eval, works with local Ollama.

Self-evolving 10-agent stack for one-person companies. Cron + eval + reflexion + skill library + PR-gated evolver. pip install solo-founder-os · 513 tests · MIT.

Collector-first engineering evidence backend for deterministic ingestion, Qdrant retrieval, cited investigation, readiness reports, workflows, evals, and Core MCP. Azure deployment scaffold; not yet production-grade.

Vue 3 best-practice agent skills for AI coding agents — Composition API, Router 5, Pinia 4, Vitest 4 — plus an MCP server and eval suite

Permission-aware AoA connector for authorized course-platform content: normalized knowledge, retrieval, evidence packets, local evals, and MCP tools.