AI Evals

CaseSolo launch AI workflows now have deterministic offline evals for the highest-risk legal work product. These evals do not call an external model. They grade candidate outputs against required legal concepts, structured fields, citations, approval checkpoints, client-safe language, and forbidden outcome promises.

Covered Suites

• deposition_analysis - admissions, impeachment issues, damages testimony, follow-up questions, citations, and attorney review. Since 2026-09-12 also covers prior-statement grounding quality: a cited cross-transcript conflict passes, the same claim without source cites fails.
• demand_generation - liability narrative, medical chronology, damages breakdown, demand amount, negotiation authority, citations, and approval gating.
• medical_chronology - dated treatment events, diagnosis, causation concerns, treatment gaps, bills, future care, citations, and review gating.
• liability_analysis - duty, breach, causation, comparative fault, defenses, evidence mapping, citations, and attorney approval.
• settlement_recommendation - risk-adjusted range, liens, fees, costs, negotiation plan, alternatives, citations, and authority approval.
• client_communication - plain-language status, next steps, careful client-safe language, citations, and attorney approval before sending.

Runbook

Run the focused eval suite:

npm run ai:evals

Run it through the shared test runner:

npm run test:run -- ai-evals

The suite is intentionally deterministic so failures mean one of three things:

• A prompt/output contract changed and the eval fixtures need an intentional update.
• A generated workflow stopped including required citations, fields, or approval gates.
• A safety regression introduced prohibited client-facing or legal-outcome language.

Deposition Grounding Pipeline (2026-09-12)

POST /api/depositions/[depositionId]/analysis grounds pressure points in case context without any model call:

1. Sibling depositions in the same case (firm-scoped, transcript-bearing) are

parsed into Q&A pairs with the shared parseTranscriptQA helper.

2. detectPriorStatementConflicts flags yes/no flips and affirm/deny

negation conflicts on overlapping topics. No LLM, no prompts — transcript text never enters a prompt, so PII cannot leak through prompt logging.

3. gatePriorStatementConflicts keeps only conflicts with complete

line+source cites whose assembled point text passes ethicsGate and Truth Guard (runTruthGuard with [S1]/[S2] spans over the cited lines). Anything else is dropped fail-closed.

4. Kept conflicts merge into contradictions, impeachment opportunities,

cross-exam questions, and pressure points with prior-statement cites (deponent, deposition id, line).

5. Task drafts stay drafts: analysis persists suggestedTasks as data only

and writes zero Task rows (pinned by test).

Implementation Notes

• Sources: src/lib/ai/evals.ts, src/lib/depositions/prior-statement-conflicts.ts, src/lib/depositions/pressure-points.ts
• Tests: src/__tests__/lib/ai-evals.test.ts, src/__tests__/lib/prior-statement-conflicts.test.ts, src/__tests__/api/depositions.test.ts
• The eval score is weighted by passing rubric checks and blocked by critical failures.
• Critical checks include missing required structured fields, missing citations when required, missing human review gates, unsafe client tone, guaranteed outcomes, and review-bypass language.
• Golden fixtures are local representative outputs, not production data.

Next Improvements

• Add production prompt snapshots as non-secret fixtures once prompt versioning is finalized.
• Store failed eval result JSON as CI artifacts.
• Add eval gates for the shared AI output envelope and review queue once all AI routes emit the envelope consistently.
• Add latency and cost benchmarks separately from semantic evals so quality and performance regressions are visible independently.