Agent Assurance (ATA) — AI Agent Evaluation Platform: Quality & evaluation infrastructure for autonomous AI agents

Overview

Bringing QA discipline to AI agents.

Traditional test automation asserts UI and API output. That is not enough for AI agents, whose behavior is probabilistic and evidence-based: a correct answer in one run does not prove the agent used the right context, called the right tools, reasoned soundly, or stayed within security and compliance limits.

ATA (Agent Test Agent) is an independent validation and audit layer for agentic systems. It records every interaction between a target agent and its model provider as auditable evidence, then runs an AI-native review that scores each interaction across 10 quality dimensions and turns the results into deployment-ready quality verdicts.

ATA reuses the rigor of banking quality work - evidence, regression, blockers, and audit trails - and applies it to LLM-agent behavior.

How it captures evidence

A capture layer at the model-provider boundary.

ATA inserts itself between the system under test and the model provider, so any OpenAI-compatible agent can route traffic through it with no code changes — only a base URL and an ATA-prefixed API key.

Target Agent System  →  ATA Gateway  →  Model Provider
                              │
                              └─ captures each interaction as an auditable Record

The gateway is intentionally narrow: it parses credentials, admits or rejects requests, forwards provider traffic, and emits evidence events. It never scores, finalizes, or runs validator logic — keeping the capture path fast and the responsibilities clean.

Architecture

A monorepo of independent services with a shared core.

ATA is built as separate deployable services over shared internal packages, so capture, control, validation, and presentation each scale and fail independently.

• API (FastAPI, SQLAlchemy async, Alembic) — the public control plane and single source of durable state; owns projects, workflow runs, analysis lifecycle, and runtime commands.
• Gateway (LiteLLM plugin) — credential admission and record-event capture at the provider boundary.
• Worker (Celery, LangChain) — finalizes captured events into durable records and runs asynchronous, per-record validation.
• Web (React, TypeScript, Vite, TanStack Query) — an operational dashboard for projects, workflow runs, records, and analysis reports.
• Shared packages — a canonical SQLAlchemy ORM plus event contracts and domain rules, consumed by both API and Worker so there is one model, not per-app copies.
• PostgreSQL + Redis — PostgreSQL owns durable state; Redis owns live runtime coordination and acts as the Celery broker. Local Compose wires every service together end to end.

Validation method

Record-first, evidence-based validation across 10 dimensions.

The evaluation unit is the record — a normalized snapshot of one provider interaction (request, context, tool calls and results, response, usage, and runtime metadata). Each record is a clean evidence package that can be audited, replayed, and regression-tested. Every record is scored against the dimensions that actually apply; missing evidence is marked N/A rather than guessed.

1. Context Quality        6. Security        (blocker)
2. Prompt Quality         7. Compliance      (blocker)
3. Tool Call Quality (b)  8. Operations
4. Reasoning Quality      9. Human Validation
5. Output Quality        10. Business Outcome

Semantic dimensions are evaluated by an LLM judge; Operations, Human Validation, and Business Outcome are deterministic. Security, Compliance, and Tool Call Quality are blocker dimensions that can stop a deployment outright.

Multi-agent validator

Planner → Judge → Critic → Meta Judge, per record.

Each record runs through a plan-and-execute validator agent system rather than a single LLM call. This adversarial, self-checking structure is what makes the scores trustworthy.

• Planner — analyzes the record's evidence and risk surface to select applicable dimensions and mark the rest N/A; it never blindly runs all 10.
• Judge — produces an initial score, rationale, and evidence references for each selected dimension against the rubric.
• Critic — attacks the Judge's output to find unsupported conclusions, ignored evidence, logic gaps, or missed risks.
• Meta Judge — resolves Judge vs. Critic disagreement into a final result and flags low-confidence or high-risk records for human review.

The judge is never treated as an oracle: when evidence is weak the Meta Judge must lower confidence or return N/A, and high-risk decisions use self-consistency and human sampling for calibration.

Scoring & quality gate

From metric scores to a deployment verdict.

Scoring is bottom-up — metric → dimension → record → workflow — under a No Average Hiding principle, so a high average can never bury a weak dimension or a critical blocker.

Metric score (0–100)
  → Record score   (0.6 × median + 0.4 × weakest dimension)
  → Workflow Quality Gate (P10/P35/P50, mean, deviation, blockers)
  → Verdict: PASS · WARN · FAIL · BLOCK

The Workflow Quality Gate is the single decision point: it judges the distribution of record scores, not just the mean, so weak low-tail records and critical blockers correctly prevent a PASS. Repeatable analysis runs and a reproduce/compare flow make quality measurable over time.

Security & data handling

Fail-closed by default; provider secrets stay request-local.

• Recording uses a scoped credential format ata-<run-tag>--<provider-secret>; the gateway fails closed for malformed, unknown, inactive, or unavailable runtime states.
• The provider secret is request-local only and is never written to Redis, PostgreSQL, logs, traces, analysis results, or UI state.
• Validator calls use separate configured credentials and can never reuse a target's recording token.
• Captured request/response payloads are treated as sensitive evidence, not public telemetry.

Engineering decisions

Trade-offs and design rationale.

• Capture-only gateway — keeping scoring out of the request path makes capture cheap and lets validation evolve independently.
• API owns state; Worker owns side effects — all state transitions flow through the API; the worker handles async finalization and validation with idempotent upserts and bounded retries.
• One shared ORM — a single canonical persistence model removes drift between the API and worker.
• Single product vocabulary — workflow_run is used end to end across DB, API, and UI, with one squashed Alembic baseline for the MVP schema.
• MVP focus — deterministic, demo-ready scoring with production calibration, streaming capture, tenancy, and continuous-learning loops deliberately deferred.

Status & Integration

A completed standalone MVP, now integrated into Omni-Agent.

ATA runs as a completed historical MVP that proved the multi-agent validation architecture. The core capabilities have since been integrated into Omni-Agent as its native evaluation module, merging validation, audit trails, and quality-gate verdicts directly into the flagship platform core.

Stack