---
title: "Error Handling"
slug: error-handling
section: Governance & Operations
status: stable
description: "Unified resilience patterns for the Globus ecosystem — failure modes, recovery strategies, circuit breakers, escalation workflows, and operational runbooks across all integration points."
last-updated: 2026-10-07
---

# Error Handling

Unified resilience patterns for the Globus ecosystem — failure modes, recovery strategies, circuit breakers, escalation workflows, and operational runbooks across all integration points.

> **Core Principle: Fail Explicitly, Recover Predictably**
>
> Every integration point in the ecosystem has a defined failure mode, a bounded retry strategy, and a clear escalation path when automated recovery fails. Silent failures are the most expensive kind of bug — the system must surface them, not swallow them.

## Unified Resilience Principles

Error handling in the Globus ecosystem follows six core principles that apply uniformly across all integration points — SEVIS, NetSuite, payment gateways, webhooks, and the event outbox.

### Existing Error Handling Patterns

Several error handling patterns are already defined across the ecosystem documentation. This page unifies them — the following are referenced, not duplicated:

### Resilience Hierarchy

When a failure occurs, the system follows a four-level hierarchy before escalating to humans:

## SEVIS Failure Modes & Recovery

SEVIS (Student and Exchange Visitor Information System) is a government dependency with strict availability constraints, batch processing windows, and no tolerance for duplicate submissions. Failures in SEVIS integration directly impact applicant lifecycle progression and compliance status.

> **Government Dependency**
>
> SEVIS is an external government system with limited availability windows, no SLA guarantees, and strict validation rules. The Globus integration must be resilient to all SEVIS-side failures without compromising applicant data integrity.

### Failure Scenarios

| Failure | Cause | Impact | Recovery |
| --- | --- | --- | --- |
| Connection Timeout | SEVIS system unavailable or network issue during batch window | Batch submission stalls; affected applicants remain in pending_sevis state | Retry in next batch window. Circuit breaker is configured to open after 5 failures in a rolling 60 sec window. |
| Validation Rejection | Applicant data fails SEVIS validation rules (e.g., invalid passport format, missing required field) | Individual applicant record rejected; rest of batch continues | Flag applicant for data correction. Staff reviews and re-submits in next batch. |
| Duplicate Submission | Retry after ambiguous response creates duplicate SEVIS record | Applicant has two SEVIS IDs — compliance risk | Idempotency key prevents duplicates. If detected, manual SEVIS correction required. |
| Partial Batch Failure | SEVIS accepts some records but rejects others in the same batch | Mixed success/failure state across batch applicants | Process successes immediately. Queue failures for individual retry with corrected data. |
| Authentication Failure | SEVIS API credentials expired or revoked | All SEVIS operations blocked across all brands | Immediate escalation to compliance lead. Circuit breaker opens. Credential renewal required. |
| Rate Limiting | Too many submissions in a short window exceeding SEVIS quotas | Submissions throttled; batch processing slows | Automatic backoff with queue-based submission pacing. Spread batches across window. |

### Retry Strategy

Batch-Level Retries

Max 3 retry attempts per batch submission / Backoff: 15 min → 1 hour → next batch window / Circuit breaker configured after 5 failures in a rolling 60 sec window / All retries use the same batch idempotency key
Record-Level Retries

Validation failures are not retried automatically / Staff must correct data before re-submission / Corrected records join the next scheduled batch / Each record has an individual externalRef for dedup
### Manual Intervention Workflow

When automated SEVIS recovery fails, the following manual workflow is triggered:

Compliance team receives Slack alert with batch ID and failure details / Failed records appear in the SEVIS Exception Queue with reason codes / Staff reviews each failed record and corrects data or flags for investigation / Corrected records are re-queued for next batch submission / If SEVIS credential issue: compliance lead contacts Department of State for renewal / All manual interventions are audit-logged with staff ID and resolution notes
### Lifecycle Stall Handling

When SEVIS integration stalls, applicant lifecycle progression is blocked at the Visa & SEVIS phase. The system handles this as follows:

Pending SEVIS Processing
## Event Outbox Failure Scenarios

The event outbox is the durable message queue that ensures Globus state changes are reliably delivered to downstream systems (n8n, NetSuite, portals). Failures in the outbox pipeline can delay integrations but must never cause data loss.

> **Outbox Pattern**
>
> Events are written to a database table transactionally alongside the state change they represent. A background worker polls the outbox and delivers events. This guarantees at-least-once delivery even when downstream systems are unavailable.

### Worker Downtime

What Happens

Events accumulate in the outbox table (durable, not lost) / Downstream systems receive no updates during downtime / Portals may show stale data but remain functional in read mode / No Globus data corruption — the outbox is append-only
Recovery

Worker restart processes the backlog in FIFO order / Backlog size is monitored — alert if > 1,000 pending events / Worker health check runs every 60 seconds / Dead worker triggers automatic restart via process supervisor
### Ordering Guarantees

### Poison Message Handling

A poison message is an event that consistently fails delivery and blocks the processing of subsequent events.

| Step | Action | Threshold |
| --- | --- | --- |
| Detect | Track per-event retry count. Flag events that have failed 5+ delivery attempts. | 5 failed attempts |
| Isolate | Move the poison event to a dead-letter table. Resume processing subsequent events. | Automatic after detection |
| Alert | Send Slack notification to the integration team with event ID, payload, and error details. | Immediate on isolation |
| Investigate | Staff examines the event payload and delivery error. Determines if it is a data issue, schema change, or consumer bug. | SLA: 4 hours |
| Resolve | Fix the root cause, then replay the event from the dead-letter table. | Manual action required |

### Monitoring & Alerting

## NetSuite Sync Failure & Staleness Detection

Financial data flows from NetSuite into Globus via scheduled sync jobs and webhook callbacks. When this pipeline fails or lags, Globus displays stale financial data. The system must detect staleness, inform users, and provide recovery procedures.

> **NetSuite Is the Financial Truth**
>
> When Globus cache and NetSuite diverge, NetSuite wins. Globus never independently updates financial records — it only caches what NetSuite reports. See Finance & NetSuite for the system-of-record boundary.

### Staleness Detection Thresholds

| Data Type | Acceptable Lag | Warning Threshold | Critical Threshold | Detection |
| --- | --- | --- | --- | --- |
| Invoice Status | < 5 minutes | 5–15 minutes | > 15 minutes | lastSyncedAt on application_finance_summary |
| Payment Status | < 5 minutes | 5–15 minutes | > 15 minutes | lastSyncedAt on finance_payment_refs |
| Customer Records | < 1 hour | 1–4 hours | > 4 hours | Last successful customer sync timestamp |
| Exchange Rates | < 24 hours | 24–48 hours | > 48 hours | Rate refresh timestamp in FX cache |

### Portal Display Rules for Stale Data

Financial data may not be current. Last updated [timestamp]. / Financial data is significantly out of date. Contact support if you need current information.
### Reconciliation Recovery Process

Detect sync failure via monitoring (lastSyncedAt exceeds critical threshold or sync job reports errors) / Pause outbound finance events to prevent further divergence during recovery / Run full reconciliation job: pull all active invoices and payments from NetSuite API / Compare NetSuite data against Globus cache — identify discrepancies in amounts, statuses, and references / Apply NetSuite-authoritative values to Globus cache (NetSuite wins for all accounting data) / Resume outbound finance events and verify sync pipeline is healthy / Generate reconciliation report for finance team review
## Payment Gateway Failures

When online payments are enabled (Phase 3+), the payment gateway introduces a third external dependency between the applicant, Globus, and NetSuite. Gateway failures can result in inconsistent state between what the applicant sees and what NetSuite records.

### Failure Scenarios

| Scenario | What Happens | Applicant Sees | System Response |
| --- | --- | --- | --- |
| Card Declined | Gateway rejects the payment attempt | 'Payment declined. Please try a different payment method.' | No state change in Globus or NetSuite. Invoice remains OPEN. |
| Gateway Timeout | Gateway does not respond within 30 seconds | 'Payment processing. Please do not submit again.' | Mark payment as PENDING_CONFIRMATION. Poll gateway for result. If no result after 5 min, escalate. |
| Partial Payment | Payment amount is less than invoice total (split payment scenario) | Updated balance showing remaining amount due | Record partial payment in NetSuite. Update Globus cache to PARTIALLY_PAID. |
| Webhook Failure | Gateway processes payment but webhook to Globus/NetSuite fails | Portal still shows UNPAID despite successful charge | Gateway webhook retry (3 attempts). If all fail, scheduled reconciliation catches it within 15 min. |
| Double Charge | User submits payment twice due to timeout/retry | 'Payment already processing. Duplicate submission prevented.' | Idempotency key on payment intent prevents duplicate charges. If duplicate detected post-facto, auto-refund triggered. |
| Refund Failure | Refund request fails at gateway level | Refund status shows 'Processing' | Retry refund 3 times. If all fail, escalate to finance team for manual refund via gateway dashboard. |

### Globus–NetSuite Consistency

When a payment fails mid-flow, Globus and NetSuite can end up in inconsistent states. The following rules ensure consistency is always restored:

## Finance Reconciliation Recovery

When application_finance_summary diverges from NetSuite truth, the system must detect, flag, and resolve the discrepancy. This section covers the reconciliation process for both automated and manual recovery.

### Divergence Detection

| Divergence Type | Detection Method | Auto-Correctable? | Resolution |
| --- | --- | --- | --- |
| Amount Mismatch | Globus amountDue ≠ NetSuite open balance | Yes | Overwrite Globus cache with NetSuite values |
| Status Mismatch | Globus shows INVOICED but NetSuite shows PAID | Yes | Update Globus financeStatus to match NetSuite |
| Missing Invoice | NetSuite has invoice but Globus has no cached record | Yes | Create cache record from NetSuite data |
| Orphaned Cache | Globus has cached invoice that does not exist in NetSuite | No | Flag for finance team review — may indicate voided invoice |
| Currency Mismatch | Globus currency differs from NetSuite invoice currency | No | Flag for finance team review — may indicate configuration error |

### Reconciliation Triggers

## Escalation Matrix

When automated recovery fails, the system escalates to humans via defined channels. The escalation matrix maps each integration point and severity to specific roles and notification channels.

### Severity Levels

| Severity | Definition | Response Time | Channels |
| --- | --- | --- | --- |
| P1 — Critical | Complete integration failure. Multiple users affected. Data loss risk. | 15 minutes | Slack (urgent), SMS, Phone |
| P2 — High | Partial integration failure. Some users affected. Degraded functionality. | 1 hour | Slack (high-priority), Email |
| P3 — Medium | Non-blocking issue. Single user or delayed processing. No data loss. | 4 hours | Slack (standard), Email |
| P4 — Low | Cosmetic or minor delay. No user impact. Informational. | Next business day | Email, Ticketing system |

### Escalation Paths by Integration Point

| Integration | Failure Example | Severity | Primary Responder | Escalation (if unresolved) |
| --- | --- | --- | --- | --- |
| SEVIS | SEVIS circuit breaker opens (configured: 5 failures in 60 sec) | P1 | SEVIS Compliance Lead | Program Director → CTO |
| SEVIS | Individual record validation failure | P3 | Ops Staff (SEVIS team) | SEVIS Compliance Lead |
| NetSuite Sync | Sync job fails for > 15 minutes | P2 | Integration Team | Finance Lead → CTO |
| NetSuite Sync | Customer creation fails | P3 | Integration Team | Finance Lead |
| Payment Gateway | Gateway completely unreachable | P1 | Integration Team | Finance Lead → CTO |
| Payment Gateway | Single payment webhook fails | P3 | Integration Team | Finance Lead |
| Webhooks (Portals) | Portal webhook delivery fails after retries | P2 | Integration Team | Platform Lead → CTO |
| Event Outbox | Worker down > 5 minutes | P1 | Platform Team | CTO |
| Event Outbox | Poison message detected in DLQ | P3 | Integration Team | Platform Lead |
| Quota Engine | Race condition causes over-allocation | P2 | Platform Team | Program Director → CTO |

## Circuit Breaker Patterns

Circuit breakers prevent cascading failures by stopping retries to an unhealthy dependency. When a circuit breaker opens, the system enters a defined degraded mode for that integration point.

> **Why Circuit Breakers?**
>
> Without circuit breakers, a failing dependency causes retry storms that exhaust connection pools, increase latency across the entire system, and can turn a partial outage into a full one. Circuit breakers let the system fail fast and recover gracefully.

### Thresholds & Configuration

| Dependency | Failure Threshold | Window | Open Duration | Half-Open Probes |
| --- | --- | --- | --- | --- |
| SEVIS API | 5 failures | Rolling 60 sec | 120 sec | 1 probe |
| NetSuite API | 5 failures | Rolling 60 sec | 60 sec | 1 probe |
| Payment Gateway | 3 failures | Rolling 60 sec | 30 sec | 1 probe |
| Portal Webhooks | 10 failures | Rolling 60 sec | 30 sec | 2 probes |
| n8n Automation | 10 failures | Rolling 60 sec | 30 sec | 2 probes |

### Open-Circuit Behavior

SEVIS processing delayed / Pay Now / Payment processing is temporarily unavailable.
## Operational Runbook Summaries

Short recovery procedures for the most common failure scenarios. Each runbook follows a consistent structure: Detect → Assess → Act → Verify → Document.

### Runbook: Stale Finance Data

Symptoms

Portal financial data shows outdated payment status. lastSyncedAt timestamp is older than 15 minutes. Finance staleness alert triggered.

Steps

Check NetSuite API health: verify connectivity and authentication via health-check endpoint / Check outbox worker status: verify the worker process is running and heartbeat is current / If NetSuite is down: confirm with NetSuite status page. Wait for recovery. Circuit breaker handles the rest. / If worker is down: restart the outbox worker process. Monitor backlog processing. / If both are healthy but data is stale: run on-demand reconciliation for affected records / Verify: confirm lastSyncedAt is updated and portal data reflects current NetSuite state / Document: log the incident, root cause, and resolution time in the ops incident log
### Runbook: Stuck SEVIS Batch

Symptoms

Applicants stuck in pending_sevis for > 48 hours. SEVIS batch shows status "processing" but no response received. SEVIS Exception Queue growing.

Steps

Check SEVIS system status: verify government system availability via status page or manual login / Check SEVIS API credentials: confirm authentication is valid and not expired / If SEVIS is down: wait for government system recovery. Notify compliance team of expected delay. / If credentials expired: compliance lead initiates credential renewal with Department of State / If SEVIS is healthy: check batch submission logs for specific error codes or validation failures / For validation failures: correct applicant data and resubmit in next batch window / Verify: confirm applicants move out of pending_sevis within 24 hours of fix / Document: log incident details, affected applicant count, and resolution in compliance log
### Runbook: Failed Webhook Delivery

Symptoms

Portal data is stale. Webhook delivery logs show repeated failures. Dead-letter queue has accumulated undelivered events.

Steps

Check portal endpoint health: verify the receiving portal's webhook endpoint is reachable / Check webhook payload format: confirm the event schema matches what the portal expects / If portal is down: wait for portal recovery. Events accumulate safely in the outbox. / If schema mismatch: identify the breaking change. Coordinate with portal team for fix. / Replay failed events from the dead-letter queue once the root cause is resolved / Verify: confirm portal data is updated and matches Globus state / Document: log the failure pattern, affected events, and resolution in integration log
### Runbook: Quota Race Condition

Symptoms

Quota count exceeds the configured maximum. Multiple applications accepted simultaneously bypassed the quota check. Partner reports they were able to submit more applicants than their quota allows.

Steps

Identify affected quota: determine which partner/program/season quota was over-allocated / Count actual vs allowed: query the database for current quota usage vs configured limit / If over-allocation is minor (1–2 applicants): coordinate with program manager to approve or reject excess / If over-allocation is significant: freeze further intake for the affected quota immediately / Investigate root cause: check for missing database locks, concurrent API calls, or caching issues / Apply fix: ensure quota checks use SELECT FOR UPDATE or equivalent locking strategy / Verify: run concurrent submission tests to confirm the race condition is resolved / Document: log the over-allocation count, affected partners, and fix applied
### Runbook: Reservation Position Race (RESERVATION_POSITION_TAKEN)

Symptoms

Two partner agents (or a partner agent and the staff console) tried to reserve the same job order position at the same instant. The losing caller received HTTP 409 Conflict with error.code = "RESERVATION_POSITION_TAKEN". Alternatively, the nightly reservation-placement-integrity-scan flagged a (job_order_id, applicant_id) group with COUNT > 1 across active reservations and placements, indicating the row lock + partial unique index defenses both leaked.

Steps — Routine 409 (no integrity flag)

Confirm in audit_log that the losing reservation request rolled back cleanly — no partial counter increment, no orphaned reservation row. / Re-fetch /api/host/job-orders/:id/capacity for the affected job order. Confirm positionsReserved + positionsPlaced reflects only the winning caller. / If a partner is asking why their applicant lost, point them at the stable error.message text and the open positions in the same job order. Do NOT attempt a manual reservation insert that bypasses storage methods. / No further action required — the row lock + 409 contract worked as designed.
Steps — Integrity scanner flagged a duplicate (P1)

Page the on-call Globus engineer immediately. A flagged duplicate means both the SELECT FOR UPDATE row lock and the partial unique index on active reservations/placements failed — a release-blocker. / Freeze the affected job order: set status to a non-bookable state via the staff console so no further writes mutate the disputed counters. / Identify the two (or more) conflicting rows: query reservations + placements WHERE job_order_id = $1 AND applicant_id = $2 AND status IN (active reservation states, confirmed/active placement states). Capture row IDs, created_at, and actor. / Decide which row is canonical: the earlier created_at wins by default; if that conflicts with a confirmed placement, the placement wins. Coordinate with the program manager before discarding either. / Reverse the loser through storage methods (release reservation / void placement) so capacity counters decrement correctly. Do NOT issue raw UPDATEs — the partial unique index will reject parallel rewrites and counters must stay consistent. / Investigate root cause: confirm the storage method actually opened a transaction, confirm SELECT FOR UPDATE was on the job_orders row (not a stale read), confirm the partial unique index is present in the live schema. Check for a hand-rolled SQL path that bypassed storage methods. / Re-run the reservation-placement-integrity-scan on demand against the affected job order. Confirm zero flagged groups before unfreezing the job order. / Document: incident ticket with the duplicate row IDs, the chosen winner, the root cause, and the code/schema fix shipped.
Reservation Error Codes

| HTTP | error.code | When it fires | Caller action |
| --- | --- | --- | --- |
| 409 | JOB_ORDER_POSITIONS_FILLED | A hire (staff or host confirm, placement create as confirmed, or transition into confirmed) would take the job order past positionsTotal confirmed placements. An over-recruited job order may hold more reservations than positions, but can be hired against only positionsTotal times. Details: jobOrderId, positionsTotal, confirmedPlacements, openPositions. The transaction rolls back. | Cancel an existing placement or raise the position count before hiring again; do not retry. |
| 409 | RESERVATION_POSITION_TAKEN | The post-lock re-check on positionsReserved + positionsPlaced < positionsTotal failed: the position was reserved or placed by another caller while this request waited on the row lock. | Surface error.message verbatim. Re-fetch /api/host/job-orders/:id/capacity before any retry. Do not auto-retry on the same position. |
| 400 | VALIDATION_ERROR (details.code PLACEMENT_DATES_INVALID) | A placement start or end date is not a valid date, or the end date is on or before the start date. Enforced in the reservation service (assertHireTermDatesOrdered, called from confirmPlacement), so every hire path shares it. details carries startDate and endDate when the order is wrong. | Correct the dates and resubmit. |
| 409 | JOB_ORDER_UNAVAILABLE | A reservation was requested against a job order whose status is paused, cancelled or rejected. details: jobOrderId, status. Any quota already taken for the request is released. | Choose another job order; do not retry until the status changes. |
| 404 | RESERVATION_NOT_FOUND | confirmPlacement was called with a reservation id that does not exist. | Verify the reservation id. |
| 404 | NOT_FOUND (details.code STAFF_HOLD_NOT_FOUND) | Staff attach of an earmark to a job order (staffJobOrderReservation) found no active reservation for the application. details: applicationId. | Confirm the application has an active reservation. |
| 409 | CONFLICT (details.code RESERVATION_HELD_BY_OTHER_HOST) | Staff hire (staffParticipantHire) of a participant whose active host-level earmark belongs to a different host organization than the job order's host. details: reservationId, holdingHostOrgId, jobOrderHostOrgId. | Release the other host's earmark before hiring onto this job order. |
| 409 | CONFLICT (details.code RESERVATION_ON_OTHER_JOB_ORDER) | Staff hire of a participant who already holds an active reservation on a different job order. details: reservationId, reservationJobOrderId, requestedJobOrderId. | Hire onto that job order, or release the reservation first. |
| 500 | RESERVATION_HISTORY_UNIQUENESS_DRIFT | A reservation status change collided with idx_reservations_active_app, meaning the database index is not the expected partial unique index UNIQUE (application_id) WHERE status = 'active'. details: index, expected. | Database drift: apply migration 20271106000000_task_2395_reservation_stale_handoff_repair. |

Cross-references: Layer 7 &rarr; Reservation Concurrency & Placement Integrity (canonical lock and 409 contract); Integration Contract &rarr; Status Code Reference (409 row); Job Orders &rarr; Capacity Counters (storage-method invariant + fix #1894 partial unique index).

### Runbook: Invalid Placement Transition (PLACEMENT_INVALID_TRANSITION)

*Status: design spec, not built as described. No `PLACEMENT_INVALID_TRANSITION` code, `POST /api/placements/:id/confirm` route or `placements.confirm` permission exists. Placements are cancelled via `POST /api/placements/:id/cancel` (permission `placement.cancel`), which returns 409 `PLACEMENT_ALREADY_CANCELLED` for an already-cancelled placement; reservation transitions return `INVALID_RESERVATION_TRANSITION`. Placement history is `GET /api/admin/placements/:id/history` (staff) and `GET /api/partner/host/placements/:id/history` (host).*

Symptoms

A Confirm or Cancel call against POST /api/placements/:id/confirm or POST /api/placements/:id/cancel (or any other Layer 7 transition surface) returned HTTP 422 Unprocessable Entity with error.code = "PLACEMENT_INVALID_TRANSITION". The error envelope echoes details.from and details.to showing the rejected pair. Distinct from RESERVATION_POSITION_TAKEN (409) — that is a race on the Reserve edge; this is a transition that is not in the matrix at all.

Immediate containment

Return the 422 response unchanged. Do NOT mutate placements.status, do NOT touch the capacity counter, do NOT insert an audit row — the contract is that none of the three side effects fires for a rejected transition. / Surface error.message to the calling user verbatim; it is applicant-safe and never leaks other applicants' identities or internal counters. / If the request came from a portal, the portal must re-fetch the placement (and its history) before any retry. Do not auto-retry the same (from, to) pair — it will be rejected again.
Root-cause checks

Stale client state: portal cached the placement before another actor (or the reservation sweeper) advanced it. Confirm by reading GET /api/admin/placements/:id/history and comparing the most recent newStatus to the from value the client sent. Most common cause — closed by re-fetching. / Race with cancel or expire: a Confirm arrived after a concurrent Cancel or after the reservation sweeper transitioned Reserved → Expired. The history will show the competing transition immediately before the 422. Closed by surfacing the new state to the user. / RBAC scope mismatch: the caller had matrix-valid intent but lacked placements.confirm or placements.cancel for this placement's brand/program/season. The error envelope returns 403 instead of 422 in that case — if you actually saw 422 with PLACEMENT_INVALID_TRANSITION, RBAC is not the cause. / Genuine client bug: the portal sent a from value that has never been valid for this placement (e.g., asking to Confirm a placement that was never Reserved). Look for missing hand-off between the portal's optimistic UI and the placement's true state. Open a portal-side ticket. / Storage-layer regression: if multiple unrelated placements suddenly throw PLACEMENT_INVALID_TRANSITION for transitions that should be allowed (e.g., Reserved → Confirmed), suspect a regression in the matrix check itself. Cross-reference the Layer 7 state-machine spec and the storage method's recent commits before doing anything else.
Verification

Re-fetch GET /api/admin/placements/:id/history and confirm there is NO new transition row written for the rejected request — the contract requires the entire transaction to roll back. / Re-fetch GET /api/host/job-orders/:id/capacity and confirm positionsReserved + positionsPlaced are unchanged from before the rejected call. / If a regression in the matrix check was the root cause: run a targeted regression test that exercises every allowed (from, to) pair from the Layer 7 transition matrix before unblocking writes. / Document: log the rejected (from, to) pair, the actor, the placement.id, the originating portal/correlationId, and (if applicable) the regression fix in the incident log.
Placement Transition Error Codes

| HTTP | error.code | When it fires | Caller action |
| --- | --- | --- | --- |
| 422 | PLACEMENT_INVALID_TRANSITION | The requested (from, to) pair is not in the allowed or system-only rows of the Layer 7 transition matrix (e.g. Confirmed &rarr; Reserved, Cancelled &rarr; Confirmed, or any transition out of a terminal state). | Surface error.message verbatim. Re-fetch the placement and its history before any retry. Do NOT auto-retry the same pair. |

Cross-references: Layer 7 &rarr; Reservation & Placement State Machine (canonical transition matrix and atomic-transaction contract); Integration Contract &rarr; Placement Transition Endpoints (the Confirm and Cancel wire surface); Integration Contract &rarr; Status Code Reference (422 row).

### Runbook: Audit Emission Failure

*Status: `PLACEMENT_AUDIT_MISSING`, `PLACEMENT_AUDIT_IMMUTABLE` and the integrity scanner referenced below are not implemented as codes or jobs. Immutability of placement audit rows is enforced in the database by the `trg_placement_audit_immutable_update` and `trg_placement_audit_immutable_delete` triggers on `audit_logs` (both call the function `assert_placement_audit_immutable` and apply to rows with `entity_type = 'placement'`).*

`PLACEMENT_AUDIT_MISSING`
Symptoms

A row in placements shows a status that is not the row that was originally inserted, but audit_logs contains no matching entityType="placement" row for that transition. The integrity scanner that checks “every placement status change has a paired audit row” raises PLACEMENT_AUDIT_MISSING. The Layer 7 history endpoint (GET /api/admin/placements/:id/history) returns a chronology with a visible gap relative to the placement's current status.

Steps

Quarantine the affected placement: place an operational hold so no further automated transitions run while you investigate. / Identify the offending code path: query the application log for the most recent transaction that mutated placements.status for the affected placement.id and confirm whether the storage layer's same-transaction audit insert was bypassed. / Verify database grants: confirm the `trg_placement_audit_immutable_update` and `trg_placement_audit_immutable_delete` triggers on `audit_logs` are still present and enabled (immutability is enforced by trigger for `entity_type = 'placement'` rows, not by role grants) and that no migration dropped them. / If the gap is from a code-path bug: open a follow-up to wrap the offending transition in the same-transaction audit emission (the Layer 7 emission contract). Until the fix ships, gate the affected route behind a feature flag. / Reconstruct the missing audit row by appending a new transition with reason_code=\"corrected\" — never UPDATE/INSERT into the past. Set actor to the operator running the runbook and note the correlationId of the original transaction. / Re-run the integrity scanner to confirm PLACEMENT_AUDIT_MISSING is cleared for this placement.id and that no other placements share the same root cause. / Verify: GET /api/admin/placements/:id/history now returns a contiguous chronology with the corrected reason_code visible. / Document: log the incident, the affected placement.id, the root-cause code path, the correction row(s) appended, and the follow-up engineering ticket in the compliance incident log. This is a P1 event because it represents a breach of the Layer 7 append-only contract.
> **Error code**
>
> PLACEMENT_AUDIT_MISSING is the canonical machine code for this class of integrity failure. It is raised by the integrity scanner and surfaced in the standard error envelope when a transition is rejected for failing the Layer 7 emission contract. The scanner's cadence, scope, alert routing, and canonical payload are documented in the Audit Integrity Scanner subsection below. Related sibling PLACEMENT_AUDIT_IMMUTABLE is raised when code attempts to UPDATE or DELETE an existing audit row. See the Layer 7 append-only contract .

### Audit Integrity Scanner (PLACEMENT_AUDIT_MISSING)

*Status: design spec, not built. No `audit-integrity-scan` job exists in BackOfficePrototype and nothing emits `PLACEMENT_AUDIT_MISSING`; the cadence, scope and alert routing below are not implemented.*

`PLACEMENT_AUDIT_MISSING`
The audit integrity scanner is the periodic backstop behind the Layer 7 same-transaction audit emission contract. For every entity in scope it joins the entity's status-change history against audit_logs (filtered to the entity's entityType) and flags any status transition that has no paired audit row. Each flagged gap raises PLACEMENT_AUDIT_MISSING (or the entity-typed equivalent) and triggers the runbook above.

| Field | Value |
| --- | --- |
| Job name | audit-integrity-scan |
| Owner | Globus Platform — Layer 7 audit & history |
| Cadence | Hourly at :20 past the hour, plus a deeper full-history pass nightly at 02:45 America/Chicago. The deep pass scans every status change ever recorded; the hourly pass scans only transitions written in the last 90 minutes. |
| Scope (entity types) | placements (status), reservations (status), job_orders (status), document_packets (status), payments (status). Each scanned entity is one whose status history is contractually paired to audit_logs under the Layer 7 emission contract. |
| Comparison rule | For every distinct status value observed on an entity row, there must exist exactly one audit_logs row with the same entityType + entityId + a transition from the prior status to that status, written in the same database transaction (matched by correlationId). Missing rows are flagged; duplicate rows are not (a separate scanner handles immutability). |
| Idempotency | Read-only scan. Safe to re-run on demand without side effects. Does NOT auto-correct gaps — correction is operator-driven via the runbook above. |
| Severity if flagged | P1 — a missing audit row is a breach of the Layer 7 append-only contract, not a transient error. Any non-zero count blocks the next release of the affected code path. |
| Alerting target | Pages the on-call Globus engineer via PagerDuty service globus-platform-p1; cross-posts a summary to #globus-alerts; opens a tracking row in audit_logs tagged source=integrity_scan, entityType=audit_integrity, action=flag with the offending entityId in metadata. Compliance is CC'd on the Slack post but is not paged. |
| Remediation SLA | Acknowledge within 15 min (P1 page). Quarantine the affected entity within 1 h (operational hold). Append the corrective audit row(s) and re-run the scanner to confirm zero flags within 4 h. File the incident report in the compliance log within 24 h. |
| Retry policy | None — read-only. If the scan itself errors (DB outage), retry once after 15 min; if still failing, escalate to platform on-call as a separate P2 ("audit integrity scanner down"). |
| RBAC | View results: Globus Platform engineers, on-call rotation, Compliance (read-only). Trigger on demand: Globus Platform engineers only. |
| Runbook | Error Handling → Runbook: Audit Emission Failure (above) |

Canonical alert payload

Every flagged gap emits exactly one alert with the shape below. Downstream consumers (PagerDuty, Slack formatter, compliance incident log) parse this envelope; field names and types are stable.

`code / PLACEMENT_AUDIT_MISSING / severity / P1 / scanner / audit-integrity-scan / scanRunId / air_2026-04-29T03:20:00Z_4f2c / scanPass / hourly / nightly-deep / detectedAt / 2026-04-29T03:20:14.812Z / entityType / placement / entityId / plc_01HXYZ... / transition / expectedAuditRow / evidence / runbook / https:// /docs/error-handling#runbook-audit-emission-failure / owner / globus-platform-p1` 
> **Cross-references**
>
> Operator playbook for any flagged gap: Runbook: Audit Emission Failure Append-only contract this scanner backs: Layer 7 &rarr; Placement Status History (append-only) Sibling scanner for active-row uniqueness (different invariant): Batch Jobs &rarr; reservation-placement-integrity-scan

## Brand-Scope & Reason-Code Error Contract

Document review and step review share one structured error contract for reason-code and brand-scope failures (QA, 2026-07 — Task #1959). Previously the two surfaces disagreed (403 vs 400 for brand mismatch) and some brand checks emitted bare 403 bodies outside the standard error envelope.

- Reason-code validation in **both** review surfaces returns structured `details` with codes `REASON_CODE_REQUIRED`, `REASON_CODE_INACTIVE`, and `REASON_CODE_BRAND_MISMATCH`, plus the relevant brand ids.
- Document review's brand mismatch changed from **403 to 400**, matching step review.
- Bare 403 bodies in document review and decision brand-scope validation were replaced with the standard forbidden envelope.
- The reviewer reason-code picker serves only active, in-brand codes for **all** callers, including god mode — locked in by acceptance tests.

### Inactive-Brand Unlock Flow

Brand edits hitting **409 `BRAND_INACTIVE_READ_ONLY`** are no longer unexplained failures: the client routes god-mode users into the existing unlock dialog, which retries the edit with `override: { confirmedBy, reason }`. All other users get a read-only error toast.

## Domain Error Codes (AppError)

Domain rejections raised through `AppError` (`server/middleware/apiError.ts`) use the standard envelope `{ error: { code, message, details, traceId } }`. The HTTP status listed is the one `statusFromCode` assigns, or the one the raising code passes explicitly.

### Host Org, Job Order, Placement and Amendment State

| error.code | HTTP | When it fires |
| --- | --- | --- |
| INVALID_LIFECYCLE_TRANSITION | 500 (default) | POST /api/partner-orgs/:id/transition (partners.manage) for a host status move the lifecycle engine rejects, or pending_verification to active, which must go through the Host Activation Gate (POST /api/sponsor/host-orgs/:orgId/activate). details: hostOrgId, from, to, optional hint. statusFromCode has no case for this code, so the default 500 applies. |
| TERMINATION_CONFIRMATION_REQUIRED | 500 (default) | Defined for a terminate request without confirmation_token (documented in code as 400); no caller found in server code. |
| TERMINATION_IS_PERMANENT | 500 (default) | Defined for moving a host off terminated (documented in code as 422); no caller found in server code. |
| INVALID_PAYMENT_LINK_HOST | 422 | NetSuite finance sync received a hostedPaymentUrl whose host is not on the approved CyberSource allowlist. details.field is hostedPaymentUrl. |
| INVALID_RESERVATION_TRANSITION | 422 | A reservation status change that is not allowed from the current status. details: reservationId, from, to. |
| JOB_ORDER_NOT_IN_DRAFT | 409 | A job order workflow action that needs an editable job order (status draft or revision_requested) is attempted on a job order in another status. details: jobOrderId, currentStatus. |
| INVALID_AMENDMENT_TRANSITION | 422 | An amendment status change that is not allowed. details: amendmentId, from, to. |
| AMENDMENT_NOT_ALLOWED_IN_CURRENT_STATE | 409 | A job order is amended while its status is not approved or partially_approved. details: jobOrderId, currentStatus. |
| AMENDMENT_ALREADY_OPEN | 409 | The job order already has an in-flight amendment request. details: jobOrderId. |
| HOST_ORG_FROZEN | 409 | A job order is created, draft-edited, submitted or amended for a host organization in suspended or probation status. details: hostOrgId, state. |
| PLACEMENT_ALREADY_CANCELLED | 409 | A cancel is requested for a placement that is already cancelled (checked inside the transaction, so slot state, counters, audit log and outbox are untouched). details: placementId. |

### NetSuite Linkage

| error.code | HTTP | When it fires |
| --- | --- | --- |
| NETSUITE_ID_ALREADY_LINKED | 409 | PUT /api/integrations/applicants/:applicantId/finance-link for an applicant already linked to a different NetSuite customer id; a concurrent-write race on the uq_fap_applicant_one index maps to the same code. details: applicantId, currentNetsuiteCustomerId, attemptedNetsuiteCustomerId. |
| NETSUITE_ID_OWNED_BY_OTHER_APPLICANT | 409 | The same route, when the NetSuite customer id is already linked to another applicant or finance party (also mapped from the uq_fap_netsuite_customer index). details: netsuiteCustomerId, ownedByApplicantId. |
| NETSUITE_SYNC_PERSISTENCE_FAILED | 500 | The NetSuite finance sync could not persist (usually a schema or connectivity problem). details: operation and, when known, externalRef, sourcePaymentId, netsuiteCustomerId, sqlState. Use traceId and the SQLSTATE to diagnose. |

### Config Authoring and Publishing

| error.code | HTTP | When it fires |
| --- | --- | --- |
| REQUIREMENT_FIELD_UNKNOWN | 422 | A job order's matching requirements use a field key that is not defined for the program. details: fieldKey, programId. |
| REQUIREMENT_FIELD_INVALID_TYPE | 422 | A matching requirement value does not match its field's data type. details: fieldKey, expectedType, receivedType. |
| MATCHING_PROFILE_NOT_PUBLISHED | 422 | A job order with matching requirements is submitted for a program/season that has no published matching profile. details: programId, seasonId. |
| CAMP_TYPE_INVALID_ENUM | 422 | campType on a host profile or partner organization contains a value outside the allowlist. details: path, offendingValues, allowedValues. |
| CAMP_STYLE_INVALID_ENUM | 422 | Same check for campStyle. details: path, offendingValues, allowedValues. |
| COMPLIANCE_RULE_LOCKED | 403 | Defined for changing a compliance-locked rule without compliance.rules.manage; no caller found in server code. details: ruleKey, requiredPermission. |
| INHERITED_FROM_PARENT | 409 | Editing a host's own agreement, contacts or documents while governance inheritance is on for that field. details: field, parentOrganizationId, parentName. |
| CYCLE_TYPE_NOT_ALLOWED_BY_BUNDLE | 422 | A program's allowedCycleTypes include values the resolved regulatory bundle whitelist does not permit. details: attempted, bundleAllowedCycleTypes, forbidden. |
| CYCLE_TYPE_NOT_ALLOWED | 422 | A season/operational cycle uses a cycle type outside the program's effective allowed cycle types. details: programId, cycleType, allowedCycleTypes. |
| PROGRAM_MODEL_TYPE_FIELD_NOT_APPLICABLE | 422 | Camp-typed fields are sent on a job order whose resolved program model type does not use them. details: jobOrderId, resolvedModelType, fields. |
| BRAND_NOT_ACCEPTING_PROGRAMS | 409 | A program is created under a brand whose status is `sunsetting` or `inactive`. details: brandId, brandStatus, brandName. |
| BRAND_HAS_REFERENCES | 409 | A brand delete is blocked by dependent records; the message advises sunsetting and force-deactivating instead. details: brandId, references (count per kind). |
| PUBLISH_BLOCKED_BY_ACTIVE_APPLICATIONS | 409 | A workflow version is published without the explicit acknowledgment while the program has active applications. details: activeApplicationCount. Publishing is allowed once acknowledged; applicants then need reconciliation onto the new version. |
| STEP_EDIT_BLOCKED_BY_PUBLISHED_WORKFLOW | 409 | A step edit targets a step referenced by a published workflow (published step content is immutable; clone the step, attach it to a draft workflow and publish). details: stepKey, programIds, programNames, workflowDefinitionIds. |
| REFERENCE_KEY_CHANGE_APPROVAL_REQUIRED | 422 | A reference template edit renames or removes in-use question keys. Resend with acknowledgeKeyChanges: true to proceed. details: templateId, blockedChanges, acknowledgeField. |
| WORKFLOW_VERSION_REQUIRED | 422 | An application is created for a season with no published or bound workflow version. details: seasonId, applicationType. |

### Access and Request Validation

| error.code | HTTP | When it fires |
| --- | --- | --- |
| APPLICANT_NOT_FOUND | 404 | The applicant id in a finance-link request does not exist. details: applicantId. |
| CAPABILITY_DENIED | 403 | Two sources. Host routes guarded by requireHostCapability or its partner-user variant when the caller's role lacks the capability in role_capabilities (details: capability, role). The partner enforcement guard when the partner's autonomy model lacks the required capability (details: partnerModel, requiredCapability, action). |
| UNKNOWN_ROLE | 400 | GET /api/rbac/capabilities with a role the capability resolver does not know. details: role. |
| OPERATIONAL_CYCLE_ID_MISMATCH | 422 | A request supplies synonymous cycle fields (for example seasonId and the operational cycle id) with different values. details: field and conflictingFields (when raised while projecting the request body, before schema parsing), or field plus the legacy and canonical names and values (when raised by the bilingual cycle-field middleware). |

### Route-Level Error Codes

 Match on the code, not the message. Where a code is listed under `details.code`, the envelope `error.code` is the generic `VALIDATION_ERROR` (400) or `CONFLICT` (409).

- `POST /api/applicants/:applicationId/recommend` (`applicant.manage`): 409 `DUPLICATE_RECOMMENDATION` (top-level `error.code`) when an active recommendation already exists for the application and host organization.
- `POST /api/applications/:applicationId/camp-basis-assessments/:basisKey/review` (`applications.manage`): 409 `CONFLICT` with `details.code` `BASIS_NOT_CLAIMED` when the basis exists but is neither `pending_review` nor `verified` (a verified basis returns `VERIFIED_IMMUTABLE`).
- The camp-basis claim, review and evidence routes: 400 `VALIDATION_ERROR` with `details.code` `UNKNOWN_BASIS_KEY` when `:basisKey` is not one of `post_secondary_student`, `youth_worker`, `teacher` or `special_skills`.
- `POST /api/applications/:applicationId/classification` (`applicant.manage`): 400 `VALIDATION_ERROR` with `details.code` `CAMP_CLASSIFICATION_DERIVED` for Camp Counselor applications, whose placement is derived from Camp Personal Information.
- `POST /api/applications/:id/j2-dependents` (`applications.edit`): 400 `INVALID_DEPENDENT_TYPE` (top-level `error.code`) when `dependentType` is not `spouse` or `child`.
- `POST /api/regulatory-bundles/:id/clone-version`: 409 `CONFLICT` with `details.code` `BUNDLE_VERSION_CONFLICT` when a concurrent clone of the same bundle key landed the same version number and the one internal retry also collided. The message asks the caller to retry.
- Partner routes return these as top-level `error.code` with HTTP 400: `MISSING_REASON` (agreement `suspend`, `terminate` and `probation` actions under `/api/admin/partners/agreements/:id/` when `reason` is blank; `POST /api/admin/partners/applicants/:applicantId/reassign-attribution` without a reason; `POST /api/admin/partners/organizations/:orgId/status-change` with a reason shorter than 10 characters, unless the change is a suspension or termination that carries a `reasonCategory`), `MISSING_STATUS` (status-change with an empty `status`), `MISSING_PARTNER_ORG` (reassign-attribution with a blank `newPartnerOrgId`), `MISSING_FIELDS` (`POST /api/admin/partners/organizations/:orgId/bulk-invite` with an empty `programId` or `seasonId`), `MISSING_USER` (host-document upload, review and delete when the authenticated user has no email), `INVALID_ORG_ID` (host-document list, requirements, categories and upload routes under `/api/admin/partners/organizations/:orgId/host-documents` when `:orgId` is not a UUID) and `INVALID_PROGRAM_IDS` (`POST /api/admin/partners/organizations` for a host organization whose `programIds` include a missing or inactive program; `details.invalidProgramIds` lists them).

## Middleware Stack

The Globus backend uses a layered middleware stack with 20+ components organized into four categories. Each middleware handles a specific cross-cutting concern and is applied in a deliberate order to ensure proper request processing, authentication, audit logging, and error handling.

### Cross-Cutting Middleware

| Middleware | File | Purpose |
| --- | --- | --- |
| Correlation ID | correlationId.ts | Assigns a unique ID to each request for distributed tracing. GET /api/audit/trace/:traceId returns the traceId, an auditUrl, the service name (globus-hub-erp) and a note, so a trace id can be used to correlate request logs, error captures and MCP tickets |
| API Versioning | apiVersion.ts | Date-based version negotiation via API-Version header |
| Input Validation | inputValidation.ts | Validates request body and parameters against Zod schemas |
| Idempotency | idempotency.ts | Prevents duplicate mutations using idempotency keys |
| Async Handler | asyncHandler.ts | Wraps async route handlers for proper error propagation |
| Error Handler | errorHandler.ts | Standardized error responses with safe error messages |
| Body Parser Error Handler | index.ts | Returns 400 `INVALID_JSON` (with `details.reason`) when the request body fails JSON parsing, and 413 `PAYLOAD_TOO_LARGE` when it exceeds the size limit |
| Pagination | pagination.ts | Cursor-based pagination helpers for list endpoints |

Coded errors returned by errorHandler.ts and idempotency.ts, all in the standard `{ error: { code, message, details, traceId } }` envelope:

| HTTP | error.code | When it fires |
| --- | --- | --- |
| 403 | COMPLIANCE_ERROR | A compliance gate (ComplianceGateError) blocked the action; details.failedChecks lists the failed checks. |
| 422 | INTAKE_WINDOW_VIOLATION | The season intake-window gate rejected the request (SeasonIntakeError); details.applicationType names the flow (for example enrollment, renewal, extension) when known, otherwise details is null. |
| 400 | INPUT_VALIDATION_ERROR | InputValidationError was thrown; details is null. Zod schema failures return 400 VALIDATION_ERROR with details.issues instead. |
| 409 | IDEMPOTENCY_CONFLICT | requireIdempotency saw the same Idempotency-Key on the same route for the same user (24-hour window) with a different request body. An identical retry replays the stored response; a missing header returns 400 VALIDATION_ERROR. |
| 503 | SERVICE_STARTING | The API startup guard (startupSurface.ts) answers any `/api/` request that arrives after the listener opens but before route registration finishes, with message "Application is starting. Retry shortly." and details null. Retry after a short delay. |

### Audit Middleware

| Middleware | File | Purpose |
| --- | --- | --- |
| Audit Mutation | auditMutation.ts | Tracks state-changing requests with before/after snapshots |
| Mutation Registry | mutationRegistry.ts | Registry of auditable mutation endpoints |
| PII Access Logger | piiAccessLogger.ts | Logs access to sensitive personal data fields |
| PII Redactor | piiRedactor.ts | Redacts personally identifiable information in logs and error responses |
| Actor Context | actorContext.ts | Establishes current actor (user or API key) context for the request |

### Audit Log (docs service)

Every mutating endpoint in server/routes.ts writes an audit_logs row through the shared recordAudit() helper after the success response is sent. Audit writes are best-effort: failures are caught and logged to stderr but never fail the underlying request, so audit emission can never bring down a live mutation. The corresponding runbook lives at Audit Emission Failure.

| Field | Type | Notes |
| --- | --- | --- |
| userId | int \| null | Real partner user id, or null when the actor is staff/system. |
| action | string | Dotted vocabulary: asana.*, mcp.*, partner_organization.*, host_organization.note.* |
| entityType | string | asana_project \| asana_portfolio \| asana_task \| mcp_node \| partner_organization \| host_organization_note |
| entityId | int \| null | Numeric entity id where applicable; null for fan-out actions (e.g. portfolio sweeps). |
| before / after | jsonb \| null | Snapshots of the changed fields. proxiedByUserId is folded into `after` when staff are impersonating. |
| timestamp | timestamp | Server clock at the moment the row was recorded. |

Staff browse the resulting log via the read-only Audit Console, which is backed by GET /api/audit with Zod-validated filters (entityType, entityId, userId, action, actionPrefix, from, to, limit, offset). The host-scoped wrapper /api/partner/host/overview/audit continues to serve the partner portal unchanged.

### Auth Middleware

| Middleware | File | Purpose |
| --- | --- | --- |
| API Key Auth | apiKeyAuth.ts | Validates and hashes API keys for external integration authentication |
| Partner Auth | partnerAuth.ts | Partner-specific authentication for portal access |
| Partner Enforcement Guard | partnerEnforcementGuard.ts | Enforces partner agreement terms and restrictions |
| Partner Scope Filter | partnerScopeFilter.ts | Multi-tenant data scoping for partner data isolation |
| Partner Type Guard | partnerTypeGuard.ts | Validates organization type matches expected partner type |
| Require Host Role | partnerAuth.ts (requireHostRole) | Enforces host user role requirements on host portal routes |
| Require Partner Capability | partnerAuth.ts (requirePartnerScope, requireHostCapability, requirePartnerUserCapability) | Fine-grained scope and capability checks for partner RBAC |
| Field Permission Enforcer | fieldPermissionEnforcer.ts | Per-field access rules on step and applicant data. Writes are refused with 403 FIELD_ACCESS_DENIED (field hidden or no access), 403 FIELD_READ_ONLY (view-only) or 403 FIELD_OVERWRITE_DENIED (a role without sponsor override writing a value last set by a different role); each carries details.fieldPermission. Reads strip hidden fields and add `_fieldPermissions` (`readOnlyFields`, `hiddenFields`, `rules`). If the check itself errors it fails closed with 500 FIELD_PERMISSION_ERROR. |

Error codes returned by the partner guards:

| HTTP | error.code | When it fires |
| --- | --- | --- |
| 400 | MISSING_PARTNER_CONTEXT | partnerEnforcementGuard could not derive a partner organization from the request. |
| 404 | PARTNER_NOT_FOUND | The derived partner organization id has no organization row. |
| 403 | PARTNER_SUSPENDED | The organization status is suspended. |
| 403 | PARTNER_TERMINATED | The organization status is terminated. |
| 403 | PARTNER_PENDING_VERIFICATION | The organization status is pending_verification (internet presence not yet verified). |
| 403 | NO_ACTIVE_AGREEMENT | No partner agreement with status active whose start and end dates span the current time. |
| 403 | LIFECYCLE_PHASE_DENIED | The action is not allowed in the current lifecycle phase; details carry currentPhase with allowedPhases (route configuration) or with action and applicantId (per-applicant phase check on non-GET requests). |
| 401 | USER_NOT_FOUND | partnerAuth received X-Partner-Org-Id with X-Partner-User-Email but no partner_users row matches that email in that organization. |
| 401 | USER_DEACTIVATED | The matching partner user exists but isActive is false. requireHostRole answers an inactive host user with 403 USER_DEACTIVATED instead. |
| 400 | MISSING_USER_EMAIL | requireHostRole was called without the X-Host-User-Email header. |

### System Middleware

| Middleware | File | Purpose |
| --- | --- | --- |
| SEVIS Rate Limiter | sevisRateLimiter.ts | Rate limiting for SEVIS API calls to prevent exceeding government quotas |
| Maintenance Mode | maintenanceMode.ts | Blocks incoming requests during scheduled maintenance windows with appropriate status responses |

> **Maintenance Mode**
>
> The maintenanceMode.ts middleware can be activated to block all incoming API requests during scheduled maintenance windows. When active, it returns a standardized 503 Service Unavailable response with a retry-after header. Critical system health endpoints remain accessible to monitoring tools.