AI Governance Metrics

God Mode exposes an AI governance scorecard at /god-mode/ai through /api/god-mode/ai. The goal is to make AI quality, safety, cost, and approval readiness visible to platform operators without relying on static status claims.

Metrics Included

• Model usage: calls, successes, failures, token volume, reported or credit-estimated cost, average latency, and last-seen time.
• Prompt versions: prompt IDs, explicit prompt versions, active config fingerprints, model, calls, and active config count.
• Eval scores: deterministic launch eval results for deposition analysis, demand generation, medical chronology, liability analysis, settlement recommendations, and client communications.
• Approval rates: pending, approved, rejected, changes requested, applied, high-risk pending, approval rate, and correction rate.
• Hallucination reports: Truth Guard unsupported-claim, contradiction, hallucination, and correction signals.
• Cited-source coverage: recent AI runs with citations or source spans, average sources per run, and the first missing-source run IDs.
• Cost and latency: total tokens, reported or credit-estimated spend, average latency, and p95 latency.

Data Sources

• Activity rows where action is ai_call, starts with ai_, or starts with ai.
• Recent AgentRun rows with sourceSpans, truthGuardReport, output, token, and duration fields
• AgentConfig prompt configuration, system prompt fingerprint, active status, model parameters, and update timestamp
• Firm settings under aiReviewQueue.requests
• Offline eval summary from src/lib/ai/evals.ts

Cost Policy

The scorecard uses reported estimatedUsd when present. If cost is not reported but creditsUsed is present, it applies the internal credit estimate of $0.01 per credit. If neither value exists, cost remains 0 rather than inventing provider pricing.

Safety Policy

Governance health is intentionally conservative. Missing citation coverage, correction requests, hallucination reports, and failed eval suites reduce the score. Critical eval failures are surfaced at suite level so operators can see whether the problem is missing citations, missing review gates, unsafe client language, or incomplete structured fields.

Verification

Run focused tests:

npm run test:run -- ai-governance-metrics ai-evals

Run targeted lint and typecheck when changing the route or UI:

npx eslint src/lib/ai/governance-metrics.ts src/__tests__/lib/ai-governance-metrics.test.ts src/app/api/god-mode/ai/route.ts 'src/app/(dashboard)/god-mode/ai/page.tsx'
npm run typecheck

Next Improvements

• Persist eval run history so God Mode can chart quality over time instead of showing only the current deterministic suite.
• Attach route-level prompt IDs and versions to every AI output envelope.
• Store correction outcomes and hallucination reports as first-class records instead of inferring from activity metadata.
• Add CI artifacts for failed eval suites and missing citation run IDs.