Runtime Operations: Jobs, Realtime, SSE, Sockets, Storage, and Uploads

This guide covers the operational surfaces that run outside ordinary request and response rendering.

Background Jobs

Job infrastructure lives in src/lib/jobs.

• Queue setup: src/lib/jobs/queue.ts
• Runtime helpers: src/lib/jobs/runtime.ts
• Worker registration: src/lib/jobs/workers.ts
• Processors: src/lib/jobs/processors/*
• Task templates: src/lib/jobs/task-templates.ts

Current processor domains include agent runs, auto-tasks, communications, document processing, jurisdiction updates, overdue detection, RAG ingestion, SOL tracking, task reminders, and workflow execution.

Job rules:

• Every job payload must include enough firm or user context to enforce tenancy.
• Processors should be idempotent or carry an idempotency key.
• Retries must not duplicate user-visible mutations.
• Processor errors should be observable and safe to replay.
• Do not put secrets in job names, logs, or payload snapshots.

Workflow Execution Queue

Workflow triggers call evaluateTriggers() in src/lib/workflow/triggers.ts. When REDIS_URL is configured, matching active workflows are handed to the casesolo-workflow-execution BullMQ queue through enqueueWorkflowExecution(); the request path receives a queued result instead of running the workflow inline. When jobs are not configured, the same trigger path falls back to synchronous executeWorkflow() so local and preview environments still work without Redis. Trigger names are normalized through src/lib/workflow/event-names.ts: new workflow records are stored with canonical emitted event names, while evaluateTriggers() still queries known legacy/webhook aliases so existing saved workflows continue to fire.

Each executed node writes a WorkflowRunStep row with node ID/type, order, input, output, status, timing, branch target, skip reason, or error. When Redis jobs are configured, delay nodes pause the current WorkflowRun, persist the downstream node in currentNode, and enqueue a delayed workflow-resume:{runId}:{nodeId} job back into casesolo-workflow-execution. Without Redis, delay nodes keep the local synchronous fallback and continue immediately.

wait_for_event nodes now pause runs with an event resume token in the wait step output. evaluateTriggers() checks paused runs for matching event tokens, case scope, and currentNode; matching runs resume inline without Redis or are handed back to casesolo-workflow-execution when jobs are configured. Event matching accepts the canonical underscore event names and common webhook/builder aliases such as case.status_changed and case_status_changed.

Wait-for-event nodes can also carry a timeoutDays, timeoutHours, timeoutMinutes, or timeoutMs value. When jobs are configured, the engine queues a delayed workflow-timeout:{runId}:{waitNodeId}:{resumeNodeId} job. If the event arrives first, the timeout job later exits because the run is no longer paused. If the timeout wins the atomic claim, the run is marked FAILED with a timeout log entry before any downstream node side effects run.

Workflow runs can be cancelled with DELETE /api/workflows/{workflowId}/runs/{runId} and an optional JSON { "reason": "..." } body. The route is firm-scoped, calls cancelWorkflowRun(), marks non-terminal runs CANCELLED, clears currentNode, writes a cancel log entry, and best-effort removes any scheduled resume job. Delayed jobs use the stored resume job ID; event-wait resume jobs derive the same stable workflow-resume:{runId}:{currentNode} ID so a queued resume can be removed before it mutates the run.

Resume workers claim paused runs with an atomic updateMany from PAUSED to RUNNING scoped by id and currentNode, so duplicate delayed or event-resume jobs cannot both advance the same node. A worker that loses the claim reloads the run state and exits without executing node side effects.

Workflow traversal now runs through an explicit stack/while-loop instead of recursive child calls. This preserves the existing depth-first edge order and pause semantics while adding path-scoped cycle detection and a traversal step budget, so malformed builder graphs fail the run with a clear Workflow traversal cycle detected error before re-entering a node or repeating side effects indefinitely.

Recoverable action problems use the engine's skip-vs-fail path. A skipped node records status: "skipped" in the run log, marks its WorkflowRunStep as SKIPPED with a skipReason, and then continues through outgoing edges. True unexpected exceptions still mark the node FAILED and fail the run. This keeps misconfigured optional actions from hiding downstream automation while preserving hard failure semantics for data mutations and infrastructure errors.

Workflow run-step realtime updates use the P3 event contract in src/lib/realtime/workflow-events.ts. The first event is workflow.run_step.updated, validated by Zod before emit, and broadcast to the firm room plus the case room when a run is case-scoped. The payload includes the workflow/run/step IDs, node ID/type, step status, order, case scope, optional input/output, branch target, skip/error detail, duration, and occurrence time. The workflow engine emits these updates as each WorkflowRunStep starts and when it completes, skips, or fails. The emit is best-effort; Prisma remains the durable source of truth if Socket.io is unavailable. Browser clients consume the same event through useSocket(), which exposes a capped workflowStepUpdates event log plus clearWorkflowStepUpdates(runId?) for run-history surfaces. The workflow builder's Run History dialog fetches /api/workflows/{workflowId}/runs?limit=10, renders durable ordered WorkflowRunStep rows returned as steps, and merges live step updates while open. Socket.io Redis fan-out is enabled when REDIS_URL is configured.

Socket.io uses the in-memory adapter by default for local/no-Redis environments. When REDIS_URL is present, initSocketServer() creates an ioredis publisher and duplicated subscriber, then attaches @socket.io/redis-adapter so firm, case, user, presence, and workflow step events fan out across web instances. Set CASESOLO_SOCKET_REDIS_ADAPTER=disabled to force the in-memory adapter while diagnosing Redis adapter issues.

Focused verification:

npx prisma validate
npm run db:migration-smoke -- --migration-history
npm run test:run -- workflow-engine jobs-workflow-execution workflow-triggers workflow-event-names jobs-queue-names workflow-variables workflow-context-builder workflow-variable-categories workflow-client workflows-route workflow-versions prisma-model-table-smoke --reporter=verbose
npm run test:run -- realtime-workflow-events socket-server --reporter=verbose
npm run test:run -- use-socket --reporter=verbose
npm run test:run -- socket-redis-adapter --reporter=verbose
npm run test:run -- workflow-runs workflow-builder-run-history --reporter=verbose
npm run lint -- src/lib/workflow/engine.ts src/lib/workflow/triggers.ts src/lib/workflow/event-names.ts src/lib/jobs/queue.ts src/lib/jobs/workers.ts src/app/api/workflows/route.ts src/app/api/workflows/[workflowId]/runs/[runId]/route.ts src/__tests__/lib/workflow-engine.test.ts src/__tests__/lib/jobs-workflow-execution.test.ts src/__tests__/lib/workflow-triggers.test.ts src/__tests__/lib/workflow-event-names.test.ts src/__tests__/lib/jobs-queue-names.test.ts src/__tests__/api/workflows-route.test.ts src/__tests__/api/workflow-run-cancel.test.ts 'src/app/(dashboard)/workflows/[workflowId]/page.tsx' 'src/app/(dashboard)/workflows/builder/_components/constants.ts' 'src/app/(dashboard)/workflows/builder/_components/trigger-section.tsx' 'src/app/(dashboard)/workflows/builder/_components/types.ts'
npm run typecheck
env CASESOLO_PLAYWRIGHT_PORT=3120 PLAYWRIGHT_BASE_URL=http://127.0.0.1:3120 npx playwright test e2e/workflow-builder.spec.ts --project=chromium --no-deps

God Mode Runtime Health

Super admins can review launch-support runtime health at /god-mode/system. The page is backed by /api/god-mode/system and includes:

• Infrastructure health for database, Redis/cache/rate-limit mode, storage, and runtime.
• API error activity derived from recent Activity rows with /api route and 5xx/error metadata.
• BullMQ queue counts for waiting, active, delayed, failed, and unreadable queues.
• Webhook endpoint counts, failed deliveries, and dead-letter delivery history from webhook metadata.
• AI run counts and recent failed agent runs.
• OCR/document processing queues, no-text results, and recent failed documents.
• Provider sync failures derived from integration/connector activity events.
• User-visible workflow failures and recent failed workflow runs.

This screen is intentionally secret-safe. It should show route names, provider names, counts, status codes, and short error summaries, but not raw credentials, tokens, prompts, document text, webhook bodies, or provider payloads.

Current telemetry caveat: API errors and provider sync failures are activity-derived. If production APM/Sentry ingestion is added later, keep this activity view as the user-support timeline and add the external incident source as a separate, labeled signal.

BullMQ counts require REDIS_URL. When jobs are not configured, the page should show the queues as unavailable or empty without treating that as a failed app request. Redis-backed rate limits may run in allow-all degraded mode when Redis is unconfigured, ping-failing, or paused after a backend failure; that state is operator-visible by design.

Focused verification:

npm run test:run -- god-mode-system --reporter=verbose
npm run lint -- src/app/api/god-mode/system/route.ts 'src/app/(dashboard)/god-mode/system/page.tsx' src/__tests__/api/god-mode-system.test.ts src/__tests__/components/god-mode-system-page.test.tsx e2e/god-mode-system.spec.ts
npm run typecheck
env CASESOLO_PLAYWRIGHT_PORT=3120 PLAYWRIGHT_BASE_URL=http://127.0.0.1:3120 npx playwright test e2e/god-mode-system.spec.ts --project=chromium --no-deps

Realtime and Sockets

Realtime modules live in src/lib/realtime.

• Socket server setup: src/lib/realtime/socket-server.ts
• Socket auth: src/lib/realtime/socket-auth.ts
• Socket Redis fan-out: @socket.io/redis-adapter + ioredis in src/lib/realtime/socket-server.ts
• Notification fanout: src/lib/realtime/notification-fanout.ts
• Collaboration provider: src/lib/realtime/collab-provider.ts
• Client hook: src/lib/realtime/use-socket.ts

Socket events must be scoped by firm, case, user, or portal session. Never trust client-supplied room IDs without verifying access server-side.

SSE

Use SSE for one-way progress streams where a full socket connection is not needed, such as AI run progress or import status. SSE routes should:

• Authenticate before opening the stream.
• Send structured event names.
• Include heartbeat comments for long operations.
• Close cleanly on abort.
• Avoid sending secrets or raw model prompts.

File Storage

S3-compatible storage helpers live in src/lib/storage/r2.ts (the file keeps its historical R2 name while supporting MinIO/S3-compatible endpoints). Upload validation lives in src/lib/upload-validation.ts.

Storage rules:

• Generate object keys with crypto.randomUUID().
• Sanitize original filenames before storing metadata.
• Validate MIME type and size before issuing upload URLs.
• Use pending records before browser PUT uploads.
• Confirm completion server-side after storage PUT succeeds.
• Hide or clean up pending and failed placeholders.
• Scope all document reads and writes by firm ID.

Upload Workflows

Staff and portal uploads follow the same safety pattern:

1. Validate request metadata and file constraints.
2. Create a pending document record.
3. Return a presigned upload URL and storage key.
4. Browser uploads the selected file to storage.
5. Client calls the completion endpoint.
6. Server confirms storage metadata and marks the record complete.
7. Failed uploads are marked failed or cleaned up.

Use focused route and component tests for both successful upload and failure cleanup. Native browser file-selection smoke is preferred for final launch evidence because in-app browser tooling may not support file upload.

Operational Checks

Before changing jobs, sockets, SSE, or uploads, run focused tests for the touched route or library. For broad runtime changes, add:

npm run typecheck
npm run lint
npm run test:run