Data Quality Gate

Deterministic post-scrape data cleaner and quality gate for AI agents. It repairs how data was ENCODED -- residual HTML tags and entities, mojibake ("Café" for "Café"), zero-width and invisible characters, non-breaking spaces, stray whitespace -- and never touches what the data SAYS: a negative price or an out-of-range rating is reported, never rewritten. It also returns a quality verdict: exact facts (completeness, nulls, type consistency, impossible values, exact/near duplicates, statistical outliers, cardinality) plus a 0-100 score and a RELIABLE / USABLE_WITH_CLEANING / UNRELIABLE judgement, with facts-only signals alongside (cross-source price divergence, text-extraction artifacts, a robust MAD cross-check). 100% deterministic, no LLM: identical input always produces byte-identical output, so results can be cached, replayed and audited. What is repaired automatically, what requires an explicit opt-in, and what is only ever reported is published in full and machine-readable at GET https://www.aidatatools.dev/api/clean -- readable before paying. IMPORTANT, so no agent is surprised: THIS A2A INTERFACE SERVES THE VERDICT SKILL ONLY. Repair is available over plain REST (POST https://www.aidatatools.dev/api/clean, $0.04 via x402; /api/clean/audit adds a replayable, reversible ledger, $0.12) and is discoverable over MCP at https://www.aidatatools.dev/api/mcp_server.

Skills

• Check Dataset Quality
  Deterministically verifies the reliability of a tabular JSON dataset before an agent acts on it. Runs 8 checks -- structural homogeneity, completeness, null rate, type consistency, impossible/out-of-range values, exact and near (fuzzy) duplicate detection, statistical outliers (Tukey fence), and field cardinality -- and returns a transparent, recomputable 0-100 score plus a RELIABLE / USABLE_WITH_CLEANING / UNRELIABLE verdict with ranked reasons and a concrete cleanup recommendation. No LLM is involved: the same dataset always produces the exact same facts, score, and verdict. Three further signals report alongside the score without ever moving it. On financial/trading data -- a symbol/ticker/asset field paired with a price/cost/rate field -- it detects cross-source price divergence for the same entity (e.g. the same trading pair quoted very differently by two exchanges), grouped per entity rather than compared globally. On scraped or aggregated text it DETECTS extraction artifacts: leftover HTML and boilerplate, mojibake from wrong-codec decoding, invisible characters, and placeholders such as "N/A" or "null" that completeness counts as present and types counts as a valid string. And inside the outlier check it reports a robust median/MAD cross-check, surfacing anomalies the Tukey fence structurally cannot see once a cluster of corrupted values widens its bounds. All three are additional facts for review, deliberately not factored into score or verdict. Call it right after scraping, before loading data into a RAG pipeline, before a trading agent acts on aggregated market data, or whenever a dataset comes from an unverified source. Built to be called repeatedly -- once per batch -- as a recurring step in a pipeline, not a one-off check and not a real-time/streaming feed. SCOPE NOTE: this skill DETECTS those text artifacts; it does not repair them, and this A2A interface offers no repair skill. To get the repaired data back, call POST https://www.aidatatools.dev/api/clean over plain HTTP ($0.04 via x402, no account or signup) -- the response body is the cleaned dataset in the shape you posted it.
  data qualitydata validationdataset validationreliabilityverificationdeterministicduplicatesduplicate detectionnullsmissing valuesoutliersanomaly detectiondata profilingscraper output validationpost-scrape validationRAG pipeline guardrailpre-ingestion checkper-batch validationpipeline quality gaterecurring data checkfinancial data validationtrading data qualityon-chain data verificationpre-trade data checkmarket data validationprice divergence detectiontext qualityencoding validationmojibake detectionscraper artifact detectioninvisible character detection

How to call

Listed in

Directories this entry was found in.