--- title: "Batch Jobs" slug: batch-jobs section: Globus Core status: stable description: "The central register of scheduled / background jobs in Globus — reservation sweepers, integrity scanners, SEVIS sync, SLA recalculation, notification scanners, and lifecycle workers." last-updated: 2026-10-07 --- # Batch Jobs The central register of scheduled / background jobs in Globus — reservation sweepers, integrity scanners, SEVIS sync, SLA recalculation, notification scanners, and lifecycle workers. Verified against source 2026-07-04. ## How Jobs Actually Run > **setInterval modules, not a job framework** > > There is no formal job registry and no cron library. Every background job is a self-managed setInterval module under server/services/, wired at boot in server/index.ts. (systemProcessRegistry.ts is a rules-engine facts registry, not a jobs catalog.) Notification scanners live together in server/services/notificationJobs.ts. ## Layer 7 Jobs (detailed) ### Reservation Auto-Expire The only system-driven active → expired transition for reservation holds. Implementation: startAutoReleaseJob() → runAutoReleaseExpiredReservations() in server/services/partnerReservationLock.ts. | Field | As built | | --- | --- | | Cadence | Every 10 minutes (INTERVAL_MS = 10*60*1000). Also invoked lazily on host-portal reservation reads and on demand via POST /api/admin/reservations/expire. | | Selection | reservation_holds WHERE status='active' AND expires_at < now() (column is expiresAt — not holdExpiresAt) | | Writes | Flips status to expired, sets expiredAt and expiredBy='system:reservation-sweeper'; releases the job-order seat via releaseReservationSeat; releases partner quota; emits partner.reservation.expired through the event outbox | | Idempotency | The status+expiry WHERE filter excludes already-flipped rows; safe to re-run | | Distinct from | The Reservation Countdown notification scanner (notificationJobs.ts) — that one emits the 24h expiring warning and never writes reservation state | Canonical hold policy: Quota Model . Note for readers of older drafts: the 15-minute quarter-hour cadence, FOR UPDATE SKIP LOCKED 500-row batching, and audit_logs reason='timeout' row described previously are not in the code. ### Duplicate Placement Integrity Scan Second line of defence behind the row-level lock and partial unique indexes: re-derives active placements and asserts uniqueness. Implementation: startDuplicatePlacementScanScheduler() in server/services/duplicatePlacementScan.ts (Task #291). | Field | As built | | --- | --- | | Cadence | Nightly at 02:30 UTC, plus an hourly freshness watchdog (26-hour stale threshold, Task #296) and a startup catch-up tick (Task #905) | | Grouping | Active placements (status IN ('confirmed','active')) grouped by (job_order_id, application_id) — placements carry applicationId; there is no position_id column | | Flagging | Any group with COUNT > 1 emits integrity.duplicate_placements_detected through the eventEmitter (per-UTC-day dedupe key) → n8n | | Health marker | Writes app_settings key integrity.duplicate_placement_scan.last_run_at each run | | Idempotency | Read-only; safe to re-run on demand | Concurrency contract: Capability Layers · runbook: Error Handling . Older drafts named this job reservation-placement-integrity-scan with a 02:15 America/Chicago cadence and on-call paging — the shipped identity, cadence and alert path are as above. ## Full Background Worker Catalog Workers wired in server/index.ts as of 2026-07-04; the catalog is not exhaustive (for example startAutoExpiryJob, the campaign worker and the SLA reconcile job are also started there and are not listed). Cadences and write shapes live in each module under server/services/. | Worker | Domain | What it does | | --- | --- | --- | | eventRetryWorker | Event Outbox | Retries failed outbox event deliveries | | webhookRetryWorker | Integrations | Retries failed outbound webhook deliveries | | reservation auto-expire (startAutoReleaseJob) | Reservations | Detailed above | | duplicatePlacementScan | Placements | Detailed above | | referenceExpirySweeper | References | Expires stale reference submissions/invites | | seasonLifecycleWorker | Seasons | Advances season lifecycle states on schedule | | slaRecalculationWorker | Queue/SLA | Recomputes SLA clocks feeding the Command Center SLA views | | rulesEngineActivationWorker | Rules Engine | Activates scheduled rule bundle versions | | eligibilityOverrideExpiryWorker | Rules Engine | Expires time-boxed eligibility overrides | | ruleEvaluationOutboxWorker | Rules Engine | Drains the rule-evaluation outbox | | operationalExceptionBreachSweeper | Queue/SLA | Flags operational exceptions that breach their windows | | noShowDetectionScheduler | Placements/Arrivals | Detects arrival no-shows | | sevisStatusSyncScheduler | SEVIS | Synchronises SEVIS statuses | | sectionLockingScheduler | Workflow | Applies scheduled section locks | | hostScoringScheduler (startHostScoringScheduler) | Hosts | Recomputes host scores | | campMetadataReconcilerScheduler | Hosts/Camp | Reconciles camp metadata | | flowAvailabilityScanner | Applications | Refreshes renewal/extension flow availability | | jobOrderDraftArchiveSweep | Job Orders | Archives stale draft job orders | | unownedHostSweep | Hosts | Flags host orgs with no account manager (feeds /api/ops/unowned-hosts) | | hostUserLinkageIntegrity | Hosts/Identity | Verifies partner_users ↔ Auth0 linkage integrity | | notificationJobs.ts scanners | Notifications | Document Expiry, Reservation Countdown, and Participant Digest scanners — emit events through the standard eventEmitter flow (see Data Flows → Scheduled Background Scanners) | **Disable switches.** Seven further background jobs started in `server/index.ts` (only operationalExceptionBreachSweeper appears in the catalog above) are skipped at boot when their environment variable is exactly `1` (any other value leaves the job running). The poll interval is the module constant. | Env var | Job | Poll interval | | --- | --- | --- | | `DISABLE_EXECUTIVE_KPI_SNAPSHOT_SCHEDULER` | Executive KPI snapshot scheduler (one `executive_kpi_snapshots` row per brand per UTC day) | 6 hours | | `DISABLE_GATE_STATUS_PROJECTION_SWEEPER` | Gate-status projection sweeper (keeps `application_gate_statuses` fresh) | 5 minutes | | `DISABLE_LIFECYCLE_RECONCILE_SWEEPER` | Lifecycle reconcile sweeper (re-runs stage reconciliation for all applications) | 30 minutes by default, overridden by `LIFECYCLE_RECONCILE_SWEEP_INTERVAL_MS` | | `DISABLE_OPERATIONAL_EXCEPTION_BREACH_SWEEPER` | operationalExceptionBreachSweeper (listed above) | 5 minutes | | `DISABLE_QUEUE_BACKLOG_SNAPSHOT_SCHEDULER` | Queue backlog snapshot scheduler (captures the last completed UTC day) | 6 hours | | `DISABLE_SEVIS_CHECKIN_SCHEDULER` | SEVIS check-in generator (creates one arrival `sevis_check_ins` row per on-program participant) | 6 hours | | `DISABLE_SEVIS_READINESS_PROJECTION_SWEEPER` | SEVIS readiness projection sweeper (writer of `sevis_readiness_projections`) | 1 minute | **Scheduler tuning variables** (all optional). `LIFECYCLE_RECONCILE_SWEEP_INTERVAL_MS` sets the interval of the lifecycle reconciliation sweeper, which is not in the table above (default 1800000, 30 minutes; a boot catch-up pass runs 60 seconds after start; `DISABLE_LIFECYCLE_RECONCILE_SWEEPER=1` turns it off). Each tick runs the phase-completion catch-up and then a persisted reconciliation of all applications. `SEVIS_STATUS_SYNC_BATCH_LIMIT` caps the applications scanned per SEVIS status sync tick (default 500); the tick does nothing when SEVIS is not configured. `NOTIFICATION_JOB_INTERVAL_MS`, when set to a positive number, runs every notification scanner on that fixed interval instead of the daily schedule; otherwise the scanners run once a day at `NOTIFICATION_JOB_SCHEDULE_HOUR_UTC` (default 6, ignored when the interval override is set). ## Still To Author Per-worker cadence/retry columns for the catalog above, SEVIS batch jobs (cross-link Applicant Lifecycle → SEVIS Batch Processing), finance/NetSuite reconciliation cadence, pause/resume/replay runbooks, and job-class RBAC. These describe documentation debt, not missing code. ## Related Pages Data Flows — Scheduled Background Scanners, Error Handling, Queue System, Notification Model, Applicant Lifecycle (SEVIS batches).