---
title: "Automation (n8n)"
slug: automation
section: Integrations
status: stable
description: "Orchestration, notifications, and integrations — without authority"
last-updated: 2026-10-07
---

# Automation (n8n)

Orchestration, notifications, and integrations — without authority

## Automation Principles

> **Core Principle: n8n Never Owns Authority**
>
> n8n is an orchestration layer. It listens for state changes in Globus and triggers side effects. If n8n goes down, Globus data remains intact, correct, and accessible. n8n is replaceable; the data it processes is not.

## What n8n Is NOT

n8n is the automation orchestration layer. It is easy to accidentally treat it as more than that. To prevent architectural drift, these boundaries are explicit:

## Safety Guardrails

> **Globus decides. n8n reacts.**
>
> n8n must never mutate canonical workflow state. Globus is the sole authority for operational decisions. n8n observes state changes and triggers side effects only.

## Why These Guardrails Matter

Without these guardrails, automation layers create race conditions (actions firing out of order) and state drift (systems disagreeing about the truth). Here are real-world examples tied to this ecosystem:

## Notification Delivery Model

Globus is responsible for generating notification events. n8n orchestrates delivery — routing applicant notifications through HubSpot and delivering partner, host, and internal notifications via SendGrid, Slack, and in-app channels. Notifications are triggered by Globus state changes and sent to applicants, partners, hosts, and internal staff. For the full notification catalog, channel routing matrix, and outbox data model, see the Notification & Communication Model .

### Durable Queue Design

All outbound notifications are stored in a persistent queue inside Globus before being processed. If n8n is temporarily unavailable:

Notifications are not lost / They remain queued inside Globus / Delivery resumes automatically when n8n is back online
### Safe Catch-Up Behavior

When processing resumes after a temporary outage:

Notifications are sent in a controlled sequence / Duplicate messages are prevented / Outdated reminders can be skipped if no longer relevant / Delivery is rate-limited to avoid message bursts
### Governance Principle

> **Globus owns the decision to notify. n8n executes delivery.**
>
> Operational workflows continue even if the automation layer is temporarily unavailable. No notifications are lost, and delivery resumes in a controlled, safe manner. Applicant-facing notifications route through HubSpot — see HubSpot as Applicant Channel .

### Notification Examples

## Notification Dispatch Architecture

Every Globus notification — applicant comms, host alerts, reference outreach, staff escalations — leaves Globus the same way: as a row in the ops_events outbox, pushed to a single n8n webhook. Globus never talks to an email/SMS provider directly. All channel selection, template rendering, and provider calls happen inside n8n. This keeps Globus authoritative and stateless about delivery, and lets the automation layer scale by adding nodes rather than endpoints.

### Single Unified Dispatcher Endpoint

Globus stores one unified Dispatcher webhook URL in integration_configs (category n8n, key webhook_url). Every event POSTs to that one URL. Routing is n8n's job, not Globus's: the Dispatcher reads the event's domain prefix (the characters before the first dot of the event type, e.g. reference in reference.invited) and hands off to the matching domain Handler, which switches on the full event type and invokes a single-purpose processor sub-workflow.

`Globus ops_events outbox │ POST (one unified webhook_url) ▼ [Globus] Dispatcher: Event Router ← reads domain prefix, fans out ├─ reference.* → [Globus] Handler: Reference Domain │ ├─ reference.invite.dispatch (processor) │ ├─ reference.reminder.dispatch (processor) │ ├─ reference.completion.notify (processor) │ └─ reference.expiry.alert (processor) ├─ application.* → [Globus] Handler: Application Domain → processors… ├─ host.* → [Globus] Handler: Host Domain → processors… └─ (unregistered)→ noOp / DLQ` 
Three layers, each independently extendable: the Router maps a domain to a Handler; the Handler maps an event type to a processor and owns DLQ wiring for its domain; the Processor does one thing — render a template, call the provider, close the loop. Adding events or domains never touches the Globus emit side or the webhook configuration.

### Dispatcher Routing Table

The Dispatcher is the single entry point for every Globus event. It exposes one webhook (integration_configs category n8n, key webhook_url), HMAC-validates the payload, and returns 202 Accepted (`{ status: "received", eventId }`) immediately — before any processing begins (ACK-first). The push succeeds the moment Globus hands off the event, and the ops_events row is released later through the event-callback. Routing then happens asynchronously: the Route Lookup node reads the event's domain prefix (the characters before the first dot) and hands off to the matching domain Handler.

| Event domain prefix | Domain Handler |
| --- | --- |
| applicant.* | [Globus] Handler: Applicant Domain |
| host.* | [Globus] Handler: Host Domain |
| reference.* | [Globus] Handler: Reference Domain |
| notification.* | [Globus] Handler: Notification Domain |
| placement.* | [Globus] Handler: Placement Domain |
| payment.* | [Globus] Handler: Payment Domain |
| document.* | [Globus] Handler: Document Domain |
| (unregistered) | noOp / DLQ |

**Registered domains at `development`.** The table above lists the original domains; the committed Dispatcher (`workflows/HYPetFZNQ3fuKhCk__globus-dispatcher-event-router.json`, Route Lookup node) registers many more. The prefix is the characters before the first dot of the event type.

| Kind | Domains |
| --- | --- |
| Routed to a handler | `applicant`, `application`, `host`, `job`, `provisioning`, `reference`, `step_review`, `eligibility_override`, `notification`, `finance`, `staff`, `netsuite`, `partner`, `placement`, `compliance`, `document`, `external_notification` (branded SendGrid delivery), `email_template` (template preview / test-send operations), and the catalog family — `catalog`, `program`, `season` (legacy alias) and `operational_cycle` all route to the Catalog Domain handler |
| Acknowledged as noOp (no handler yet, or deliberately none) | `documents`, `screening`, `step_status`, `sevis`, `host_verification`, `sla`, `capacity`, `extension_request`, `queue`, `job_order`, `housing_lead`, `brand`, `system`, `integrity`, `reference_submission`; `step` (its push handler was confirmed dead and deactivated 2026-09-02) and `workflow` (workflow.published is delivered directly by Globus to the portals, not through n8n) |

An event with an empty domain is acknowledged as a noOp. A non-empty domain that is not registered is **not** silently dropped: Route Lookup throws `Unregistered domain`, surfacing as a failure (and a failed callback), so a new Globus domain must be added to the table or explicitly registered as noOp. The table above lists `payment.*`, but no `payment` domain is registered in the committed Dispatcher (finance events use `finance` and `netsuite`).

> **Reconcile applicant.* vs application.***
>
> The illustrative tree above uses application.* for the application-lifecycle domain (see the Enriched application.created Payload below), while the live Route Lookup node registers the applicant domain as applicant.*. Confirm the canonical prefix against the deployed Dispatcher before treating either as authoritative.

> **2026-09-16 incident — one committed Dispatcher file, per-environment handler ids, and why hardcoding one literal per domain silently breaks whichever environment it doesn't belong to.** A real dev `applicant.created` event failed with `"Workflow does not exist."` The routing table's `applicant` entry had pointed at a real handler workflow since May 2026, but that id was never recorded in `workflows/id-map.json` and had no committed file anywhere — an untracked orphan sitting alongside two OTHER lookalike "Applicant Domain" copies on dev, with no durable record of which one was actually live-routed. Once found, that handler's own internal "Call HubSpot Delivery" sub-workflow reference turned out to be hardcoded to the qa/prod/sandbox-shared id instead of dev's own distinct one — a second, deeper copy of the identical mistake. The same wrong-shared-id bug was found and fixed the same day in three OTHER live dev handlers (`placement`, `compliance`, `partner`) that were silently carrying it too. Root cause: this Dispatcher file is deployed **identically** to dev, qa, prod, and sandbox, but its routing table hardcoded single literal handler ids per domain — safe only for domains where every environment happens to share the same id, and silently wrong for the handful where dev's real id genuinely differs. A prior manual live fix for four of these domains was never committed, so it was silently reverted the next time this file was redeployed from the stale source — this is the recurrence mechanism, not a one-off mistake.
>
> **Fixed 2026-09-16** (see [n8n-workflows PR #480](https://github.com/Alliance-Strategies/n8n-workflows/pull/480)): the routing table now resolves the affected domains' handler ids at runtime via a `HANDLER_ID_BY_ENV` table keyed by `$env.N8N_HOST`, so one committed file is correct in every environment simultaneously instead of drifting. `scripts/validate_references.py` gained a 4th check that recursively inspects each routing-table-referenced handler's OWN Execute Workflow nodes (this is what would have caught the HubSpot/SendGrid/Staff-domain bugs before they shipped), and `deploy-dev.yml` now runs that validation **before** deploying, not only after — a broken commit is rejected before it can overwrite known-good live content, instead of only blocking a tag advance after the damage is already done. See `workflows/dev-legacy/README.md` for which dev-only handler copies are genuinely live versus genuinely retired as of that incident — do not infer liveness from a workflow's name or which folder it's tracked in; check the live Dispatcher and `scripts/validate_references.py --env dev` directly.

**Per-environment handler ids, as committed now.** `HANDLER_ID_BY_ENV` has since been extended beyond the five domains named above: it now also covers `email_template`, `external_notification` and the catalog family (`catalog`, `program`, `season`, `operational_cycle` share one entry), each found the same way — a literal dev-only id that silently broke QA/prod/sandbox when the one Dispatcher file was promoted. The environment is resolved from `$env.N8N_HOST` (host containing `sandbox` → sandbox; starting `qa.` → qa; exactly `n8n.allianceabroad.com` → prod; anything else → dev), and the table carries a distinct prod id for `email_template` and `external_notification` (a null prod id for `email_template` was the root cause of a test-send failure, since fixed). The `email_template` and `external_notification` handlers also restrict who may execute them with n8n's `callerPolicy: workflowsFromAList`, listing the Dispatcher workflow ids, so a Dispatcher whose id is not in that list cannot call them.

Every event carries the same envelope, HMAC-signed and validated by the Dispatcher before routing:

`event_type / applicant.created / entity_id / data / timestamp / 2026-06-10T14:03:11Z` 
### Workflow Publish Events via n8n Dispatcher

The Workflow Studio publish events — workflow.published and workflow.config_published — now flow through the same managed n8n_dispatcher subscription as every other domain. They are no longer delivered by a bespoke direct-delivery path; routing, HMAC signing, ACK-first acknowledgement, and retry are all handled by the unified Dispatcher described above.

> **One delivery path for every domain.**
>
> Folding the workflow publish events into the Dispatcher removes the last special-case delivery route. Operators debug workflow-publish delivery the same way as any other event: through the Dispatcher's routing, callbacks, and delivery-health dashboards — not a separate direct-webhook code path.

**Removed direct-delivery wiring:** The `DIRECT_DELIVERY_DOMAINS` allowlist that previously let the workflow domain bypass the Dispatcher is now empty. The `APPLICANT_PORTAL_WEBHOOK_URL` environment variable is no longer consumed by the delivery layer. Any portal or automation that previously relied on direct delivery of these events must consume them from the n8n Dispatcher going forward.

**Migration backfill for long-lived environments:** A one-time idempotent data migration backfills both `workflow.published` and `workflow.config_published` into pre-existing n8n_dispatcher subscriptions so environments that existed before this change keep receiving them without requiring a re-subscription.

### Managed-Row Fan-Out Deduplication

The n8n event dispatcher previously double-delivered certain managed-row events: once via the managed webhook route and again via the generic fan-out sweep. The generic fan-out now **excludes managed rows** from its sweep, so each managed-row event reaches n8n exactly once (Fixed Aug 11 2026).

> **Managed rows are delivered once, via their managed route.**
>
> Managed-row events are owned by the managed webhook route; the generic fan-out no longer re-picks them up. This eliminates the duplicate delivery without changing which events reach n8n — only how many times each arrives.

### Scaling: Adding a New Domain

Because every event arrives at one endpoint, onboarding a new event family is purely additive inside n8n and decomposes into three independent units of work:

> **noOp routing keeps the Globus queue clean.**
>
> When Globus emits a domain n8n does not yet handle (e.g. host_verification.*, step_review.*), the Dispatcher registers it as a temporary noOp so events are acknowledged and cleared from the Globus processing queue rather than erroring and getting stuck. A real handler can replace the noOp later without any Globus-side change. Workflows are named with the [Globus] * convention so the delivery-health and workflow-health dashboards pick them up automatically.

### Dual-Callback Completion Loop

The Dispatcher closes each ops_events row with a single call to `POST /api/n8n/event-callback` (session or API key with `integrations.events`), made by its Callback: Completed and Callback: Failed nodes. The body carries `eventId`, `status` (`completed` or `failed`) and an optional `error`; `result` is additionally required when a `finance.payment_import_requested` event completes. The route stamps `processedAt` on the ops_events row, plus `completedAt` on completion and `lastError` on a failure that carries an error. An unknown `eventId` gets a 200 ack with status `unknown`, an event already `completed` or `dead_letter` is returned unchanged, and an invalid body gets 400.

The event body Globus pushes to n8n can carry a `callbackUrl` field. On a pool slot it is that slot's own origin (`deploymentOrigin("globus")`), which wins over the integration_configs row; elsewhere it is the `callback_url` value (category `n8n`) from integration_configs. The field is omitted when no value is set.

> **Notification sends add a further acknowledgement.**
>
> The callback above closes the ops_events outbox row. Notification deliveries additionally report the per-channel send result back to the notification outbox via PATCH /api/notifications//delivery-status, advancing that record from sent to delivered — see Delivery Acknowledgement Loop .

### Dispatch Failure Modes

**Handler failures are reported by the Dispatcher, not by the handlers.** The Dispatcher's Invoke Domain Handler node calls each domain handler as a sub-workflow with its error output enabled (continue-on-error). The Staff, Compliance, Partner and Placement domain handlers (workflow IDs `Wot4s3AW0DsWHr3o`, `0aSHGPIz4BWiCnls`, `DhbwK0j1xcqqZabP`, `GCuIG9MP6YcVY87X`) each still end with a "Globus Event Callback" code node, but that node no longer calls Globus itself. When a dispatch result reports a failure it throws, which fails the handler execution; the Dispatcher's error branch (Handle Handler Error, then Callback: Failed) then sends `POST /api/n8n/event-callback` with `status` `failed` and an `error` string (the thrown message, cut to 500 characters), using the Dispatcher's header-auth credential. When nothing failed the handler completes normally and the Dispatcher's Callback: Completed closes the row as described above. What counts as a failure: Staff throws `SendGrid dispatch failed (provider_status=...)` when its SendGrid Delivery result has `success: false`. Compliance throws `SendGrid dispatch failed for N/M recipient(s)` when any of its resolved recipients failed. Partner throws when a result has `success: false`, or when the partner email or the staff copy failed (unless that send was skipped). Placement throws when a result has `success: false`, when the HubSpot update failed (`hubspot_ok: false`), or when the host or partner email failed (unless skipped), so a HubSpot failure on its own now fails a Placement event.

**Earlier verb and credential mismatch (resolved).** Previous versions of the four handler callback nodes sent `PATCH /api/n8n/event-callback` with no API key, but that route is `POST`-only and authenticated, so the call could never land, the error was swallowed, and every event stayed `completed` whatever the send result. The handlers now fail the execution instead and let the Dispatcher's authenticated `POST` carry the failure (n8n-workflows #660). The Host Domain handler (`1oNUgxEtjFljL83Q`) has no callback node at `development`; it passes the SendGrid Delivery result straight to its return node, so it sends no callback of its own.

### Channel Split: HubSpot vs ESP

The dispatcher routes to different delivery channels by recipient type. Applicants are CRM contacts, so their comms flow through HubSpot (see Flow 3 below). Non-CRM recipients — references, host/ops mailboxes — are not contacts and are transacted through the configured ESP (e.g. SendGrid). Provider credentials live in n8n's environment, never in Globus.

### Delivery Workflows & Payload Contracts

Domain Handlers never call a provider SDK inline. They invoke shared, single-purpose Delivery Workflows on a fixed payload contract, so provider credentials and retry logic live in one place per channel. The committed Delivery workflows at `development` are HubSpot Upsert (before any HubSpot API call it returns a successful `skipped` result, with `skip_reason` `placeholder_email` or `test_email`, when `contact.email` ends with `@globus.placeholder`, ends with `.invalid`, or contains `@smoketest.` or `@test.`; an empty or malformed email throws), NetSuite Contact Upsert, NetSuite Customer Ensure, NetSuite Invoice (Globus), SendGrid Email, the SendGrid Event Webhook Receiver and a Keycloak create-applicant-user delivery (webhook path `keycloak-applicant-created`; it acts only on `applicant.created`, skips when a user with that exact email already exists in the `applicant-portal` realm, and otherwise creates the user enabled with no password and the Globus applicant id in a `globusApplicantId` attribute, then PATCHes the Globus applicant's `portalUserId` with the new Keycloak user id; it does not call Keycloak `execute-actions-email` and sends no email of its own), alongside the SendGrid Template Admin provider adapter and the Email Template Operation and External Notification Domain handlers described below. The proposed participant placement events that would flow through this dispatcher — and which ones lack a handler today — are catalogued in [Participant Placement Journey & Notifications](/docs/participant-placement-notifications).

> **SendGrid Email is live in production for the Reference domain only.**
>
> Corrected 2026-09-08 (previously said sandbox-only — that was stale), and narrowed again 2026-10-01. The [Globus] Delivery: SendGrid Email workflow (`xxZgUmutMOKLlrsA` in the committed production snapshot) is called directly by the Reference Domain handler (referee invite/reminder/completion emails). The Applicant Domain handler does not call it in production: the live production Applicant Domain handler (`MUdORjJIsdInSSGF`; a direct production n8n API check reported on 2026-10-01 matches the committed production snapshot) has only When Called by Another Workflow, Prepare Applicant Data, Call HubSpot Delivery, Merge Delivery Results, Extract Delivery Summary and Return Handler Result, and Prepare Applicant Data connects only to Call HubSpot Delivery. An earlier version of this note said the SendGrid path was verified live for the Applicant handler in Dev, QA, Prod and Sandbox alike; that is not true for the Applicant handler in production. The Notification Domain → HubSpot channel remains the path for other notification.* sends; SendGrid is the real, current production channel specifically for Reference-request email; applicant onboarding/welcome email now goes through the branded external_notification flow described below.
>
> **Sender identity is caller-parameterized, not a single hardcode.** The shared workflow's "Send Email via SendGrid" node defaults From/Reply-To to noreply@allianceabroad.com / "Alliance Abroad" unless the calling workflow's payload explicitly sets from_email / from_name (and optionally reply_to_email). The Reference Domain handler explicitly overrides to noreply@wildpacks.com / "WildPacks" for the referee invite, reminder and applicant completion-confirmation emails (its internal ops alerts use the generic default); the Applicant Domain handler no longer sends email at all; any other caller (e.g. the Finance handler's payment-link email, not yet live) gets the generic Alliance Abroad default. Do not reintroduce a single global hardcode here — this workflow also serves non-WildPacks domains, and a global override previously caused a real production defect.
>
> **Idempotency claim before send (dev copy only).** The dev copy of the workflow (id `1JNLQbaPPjwLWdhs`, tracked under `workflows/dev-legacy/`) claims a key before every send. After Normalize Email Input, a Claim Idempotency Key node calls `POST /api/n8n/notifications/claim` on Globus with the `eventId` and a `messageType` (the event type, or `unknown` when absent), Evaluate Claim reads the answer, and Already Sent? routes a duplicate to a Return: Duplicate Suppressed result (`success: true`, `skipped: true`) instead of to SendGrid. The endpoint inserts the `(event_id, message_type)` pair into `notification_dispatch_locks` with `ON CONFLICT DO NOTHING`, so only the first caller receives `claimed: true`. The check fails closed: any response other than `claimed: true`, including a claim-endpoint outage or a 400, is treated as a duplicate and the send is skipped. A suppressed send still returns `success: true`, so the calling handler reports it as dispatched. This is aimed at the duplicate emails caused by the at-least-once redrive of events stuck in `processing`. The committed `workflows/xxZgUmutMOKLlrsA__globus-delivery-sendgrid-email.json` on `development` carries no claim step, so that duplicate-send exposure is not closed there.

name
### Branded SendGrid Delivery (external_notification, email_template, Template Admin)

A second, Globus-governed SendGrid path sits beside the generic SendGrid Email delivery workflow. Globus keeps the template, sender identity and suppression-group decisions; n8n claims the work, fetches an instruction, calls SendGrid and reports the result.

- **`external_notification` events → [Globus] Handler: External Notification Domain.** The handler reads `payload.deliveryId`, claims the delivery (`POST /api/n8n/deliveries/:id/claim`; losing the claim race is a no-op), fetches the send instruction (`GET /api/n8n/deliveries/:id/instruction`), builds a SendGrid dynamic-template send (recipient, sender, `template_id`, dynamic template data, custom args, and an integer `asm.group_id` when an unsubscribe group is configured), posts it to SendGrid `/v3/mail/send`, and reports back: `POST .../acceptance` with the provider message id on a 202, or `POST .../dispatch-result` with `failed` (`PROVIDER_REJECTED`, or the instruction's own refusal reason) or `ambiguous` (`UPSTREAM_ERROR`). Recipients containing `sandbox-fixture@` skip the real send and simulate a 202.
- **`email_template` events → [Globus] Handler: Email Template Operation.** Same claim / instruction / result shape against `/api/n8n/template-operations/:id/{claim,instruction,result}`. A `preview` operation updates the template version's `test_data` in SendGrid; any other kind (test-send) activates the version, sends to the requested recipient and posts the outcome (`succeeded` with a provider message id, or `failed` with `provider_rejected`). An operation already finished when the execution runs is a no-op.
- **[Globus] Provider Adapter: SendGrid Template Admin.** Three webhook actions the Globus template control plane calls: `globus-adapter/template-draft` (creates a dynamic template when no `providerTemplateId` is supplied, then a version created with `active: 0` (SendGrid activates the first version of a brand-new template regardless, so the version stays inactive only when a `providerTemplateId` was supplied) with subject, HTML, plain text and test data, returning the template id, version id and a SHA-256 content hash), `globus-adapter/template-activate` (activates a template version) and `globus-adapter/template-fetch` (returns a version's subject, HTML, plain text, active flag and update time). In the committed `development` file all three adapter webhook nodes have `authentication` set to `none`, with no header-auth credential bound (the sandbox test workflow documents a header-auth harness credential bound on the sandbox instance only).
- **[Globus] Delivery: SendGrid Event Webhook Receiver.** Receives SendGrid's Event Webhook at `globus-sendgrid-events`, forwards the raw bytes and SendGrid's signature and timestamp headers unchanged to Globus (`{GLOBUS_BASE_URL}/api/webhooks/sendgrid/events`), and answers SendGrid `200` only when Globus accepted the event, `502` otherwise so SendGrid retries. In parallel it mirrors delivered, bounce, dropped and deferred events carrying a `glb_key` onto HubSpot email objects (best-effort); opens and clicks are never mirrored. The receiver's webhook node is committed with `authentication` set to `none`, so authenticity rests on Globus verifying the SendGrid signature it is passed.
- **Hardcoded welcome email removed.** The Applicant Domain handler no longer sends a hardcoded WildPacks HTML welcome email on `applicant.created` (removed on dev, then promoted to qa/prod); it now only writes the HubSpot portal-invite fields (invite URL on `applicant.created` and `applicant.invite_resent`, onboarding status, portal user id), and the welcome email is sent through the branded `external_notification` template flow. The committed Applicant Domain handler (Prepare Applicant Data node) handles only `applicant.created`, `applicant.invite_resent` and `applicant.updated`, and its only delivery call is the HubSpot delivery workflow; it makes no SendGrid call.
- **Sandbox differs from production (reported 2026-10-01, not yet in the committed snapshot).** The live Sandbox copy of the Applicant Domain handler (same workflow id, different n8n instance) was reported to have its own Prepare Email Payload and Call SendGrid Email Delivery nodes, calling the SendGrid Email delivery workflow (`xxZgUmutMOKLlrsA`) to send a hardcoded WildPacks HTML email for `applicant.created` and `applicant.invite_resent`, bypassing the branded template registry. The committed Sandbox snapshot (`workflows/sandbox/MUdORjJIsdInSSGF__globus-handler-applicant-domain.json`, last updated 2026-09-29) does not contain those nodes (it has HubSpot and NetSuite delivery calls only), so this path exists only in the live Sandbox instance and is undocumented in the repository. Production does not have it. Treat the SendGrid onboarding/invite-resend email from the Applicant handler as Sandbox-only.

### Event Flow Sequences

End-to-end traces through the three architectural layers. The Entry layer is the Dispatcher, the Domain layer is the per-domain Handlers, and the Delivery layer is the shared Delivery Workflows.

Flow A — applicant.created

`Globus — applicant.created ─▶ [Entry] Dispatcher: Event Router [Domain] Handler: Applicant Domain ├─▶ [Delivery] HubSpot Contact Upsert ├─▶ [Delivery] NetSuite Contact Upsert (no direct email send — the welcome email is sent by the branded external_notification flow)` 
Flow B — notification.send_requested

`Globus — notification.send_requested ─▶ [Entry] Dispatcher: Event Router [Domain] Handler: Notification Domain ├─ HubSpot Contact Lookup ├─ Channel Selection (WhatsApp ▶ Email ▶ SMS) └─ [Delivery] HubSpot Send Delivery ACK ─▶ PATCH /api/notifications//delivery-status ─▶ Globus` 
Mermaid source (kept inline so the diagram can be regenerated as the architecture evolves):

`sequenceDiagram participant G as Globus participant D as Dispatcher (Entry) participant H as Domain Handler participant DW as Delivery Workflow participant HS as HubSpot %% Flow A - applicant.created G->>D: applicant.created (202 ACK-first) D->>H: route applicant.* (Applicant Domain) H->>DW: HubSpot Contact Upsert H->>DW: NetSuite Contact Upsert %% no direct SendGrid send from the Applicant handler %% Flow B - notification.send_requested G->>D: notification.send_requested (202 ACK-first) D->>H: route notification.* (Notification Domain) H->>HS: Contact Lookup H->>HS: Send (WhatsApp > Email > SMS) HS-->>G: PATCH /api/notifications//delivery-status` 
## HubSpot ↔ Globus Integration

HubSpot is the system of engagement for applicants; Globus is the system of record. n8n is the broker between them. Each direction is a discrete, idempotent workflow conforming to Integration Contract v1 (one handler per operation). Every payload is HMAC-validated, every cross-system write is keyed on a stable identifier (globus_application_id, globus_applicant_id), and every flow posts a delivery confirmation back to its source.

### Four-Flow Bidirectional Event Contract

Four production workflows define the HubSpot ↔ Globus contract. Flows 1 and 4 carry data inbound to Globus; flows 2 and 3 carry data outbound to HubSpot. For the canonical Contact, Application, Program, and Season object definitions these flows read and write, see the HubSpot Integration data model.

Flows 2 and 3 close the loop by posting a delivery acknowledgement back to Globus via PATCH /api/notifications//delivery-status. For the delivery-status state machine and audit trail behind that endpoint, see Delivery Status Tracking .

### Program / Operational Cycle Catalog Sync

A scheduled (daily) workflow plus webhook listeners on program and operational-cycle change events keep HubSpot's Program and Season custom objects in sync with the Globus catalog. Programs upsert on globus_program_id; cycles upsert on globus_season_id and associate to their parent Program. Records absent from the Globus response are marked inactive in HubSpot rather than deleted.

**Event-driven path (committed at `development`).** The Dispatcher routes `program.*`, `season.*` (legacy alias), `operational_cycle.*` and the `catalog` domain to [Globus] Handler: Catalog Domain, which switches on the event type and upserts the matching HubSpot custom object: programs on `globus_program_id`, operational cycles and the legacy `season.*` events alike on `globus_operational_cycle_id` (the legacy season route maps to the operational-cycle object, not a separate season object), and countries (`catalog.country_created` / `catalog.country_updated`) on `globus_country_code` (the ISO alpha-2 code). Each upsert searches the HubSpot object by that id field and then creates or patches it; cycles are then associated to their parent Program. A deleted season or cycle is written as status `cancelled` rather than archived, because HubSpot custom objects cannot be hard-deleted. Until 2026-09-29 the Dispatcher acknowledged `program.*` as a noOp, so program events never reached this handler.

**HubSpot custom object type ids are resolved per environment.** dev, qa and sandbox share one HubSpot portal (a sandbox, portalId 51221012) while prod is a separate portal (portalId 6768913) in which every custom object has a different auto-generated type id, so a hardcoded `objectTypeId` that works on dev silently fails on prod (the 2026-10-01 incident: prod handoffs failed with `Invalid object or event type id`). `workflows/hubspot-object-type-map.json` is the canonical per-environment mapping (the same pattern as `id-map.json`), and affected nodes embed a small resolver keyed off `$env.N8N_HOST` with the same rules as the Dispatcher (host containing `sandbox` is sandbox, starting `qa.` is qa, exactly `n8n.allianceabroad.com` is prod, anything else is dev) because workflows cannot read the repo at runtime; the file and the embedded literals must be updated together. The Catalog Domain handler computes the ids once in a first Code node, "Resolve HubSpot Object Type IDs", because `$env` resolves reliably only inside Code nodes. Mapping (dev, qa and sandbox are identical):

| Object | dev / qa / sandbox | prod |
| --- | --- | --- |
| application | 2-59396507 | 2-65515441 |
| application_document | 2-59396510 | 2-66250331 |
| program | 2-62092723 | 2-66243449 |
| operational_cycle | 2-62709140 | 2-66255157 |
| country | 2-63632118 | 2-66249648 |

The Catalog Domain handler also resolves the Program-to-Operational-Cycle association type id per environment (205 on dev, qa and sandbox; 291 on prod). The org uses operational_cycle going forward: the Catalog handler's `Map: season` node now maps legacy `season.*` events onto the operational_cycle object and `globus_operational_cycle_id` rather than a distinct season object.

### Webhook Configuration Ownership

Each flow's trigger is configured on a different system. This split determines who owns each webhook registration and where to look when a flow stops firing.

| Flow | Trigger Source | Webhook Configured In |
| --- | --- | --- |
| Flow 1 — Pre-qual handoff | HubSpot Application property change | HubSpot |
| Flow 2 — Metadata sync | Globus application.* events | Globus (integration_configs) |
| Flow 3 — Notification routing | Globus notification.send_requested | Globus (integration_configs) |
| Flow 4 — Inbound reply | HubSpot conversation.message_received | HubSpot |
| Catalog sync | Schedule + Globus program/cycle events | n8n schedule + Globus |

**Flow 1 webhook authentication.** The Flow 1 trigger (`HubSpot Application Webhook`, path `hubspot-application-qualified`) is committed with n8n Header Auth and a bound header-auth credential (`HubSpot Webhook Auth (Dev)` in the committed file; the change was applied to dev and QA). The header name and secret value are held in that n8n credential, not in the repo, so the committed file does not show them. The downstream "Validate HubSpot Signature" code node only logs whether a signature header is present and passes the request through. The pre-merge `validate_credential_declarations.py` check now also fails a changed active workflow whose Webhook trigger declares `headerAuth`, `basicAuth` or `jwtAuth` without a matching bound credential.

**Flow 1 operational-cycle resolution.** The `Extract Operational Cycle ID` node resolves the cycle from the Globus operational-cycles lookup (queried with the Application's `program_code` and `status=open`). Because that endpoint can return cycles outside the requested status filter, the node also filters the returned items itself: it keeps only cycles whose program code equals the Application's `program_code` and whose status is `open`, and uses the first match. If none qualifies it throws an error containing `No open operational cycle matched programCode=` followed by the code and an `Available:` list of that program's cycles as code:id:status, so the run fails instead of provisioning against a non-open cycle.

### Pre-Qualification Eligibility Engine

Before an Application can hand off to Globus (Flow 1), an n8n eligibility engine evaluates it against the HubSpot-owned pre-qualification rules — the Program Ban List and Program Date Window objects. The engine reads rules and Globus-locked master data (Program, Country) read-only and writes only the outcome back onto the HubSpot Application.

This is the single carve-out to "n8n is not a rules engine": it runs pre-handoff, against HubSpot-owned rules, before a Globus applicant record exists — so the evaluation cannot live in Globus. In-platform eligibility and compliance logic (on-program, SEVIS, quota) remains Globus-owned. n8n never writes canonical Globus state here; it only writes the outcome onto the HubSpot Application.

Trigger: HubSpot Application property change — qualifier responses completed, or country_of_residence / target program_id set or changed. Re-runs idempotently; the latest evaluation wins.

Resolve the target Program and the Application's country_of_residence → Country association. / Ban check: if an active Program Ban List record matches (program_id + country_id, within effective_from/effective_to), set prequalification_state = disqualified with the ban_reason. / Date-window check: for each window_type, the relevant Application/applicant date must fall inside at least one active Program Date Window (active windows of a type are OR-combined). No matching window → disqualified. / Qualifier checks: evaluate remaining qualifier-response rules. All pass → prequalification_state = qualified. / Write prequalification_state (qualified | disqualified | needs_review) and a machine-readable reason back onto the Application.
Outcome qualified is the trigger condition for Flow 1 (pre-qualified applicant handoff).

> **Error handling: fail safe, never silently qualify.**
>
> Missing master data (no Country association, unmatched Program, or zero configured windows for a required window_type) resolves to needs_review, not qualified — a human decides. Transient errors (rule lookup failure, timeout) leave the prior state untouched and queue the Application in ops_events for retry. The engine never deletes or edits rules or master data; it only writes the outcome onto the Application.

### Operational Cycle Vocabulary Migration

> **season.* is deprecated in favor of operational_cycle.***
>
> Globus is renaming "Season" to "Operational Cycle" across its webhook and REST contract. During the deprecation window Globus dual-publishes: the same record emits on both season.* (envelope flagged deprecated: true) and the canonical operational_cycle.* topics, and every payload carries both seasonId and operationalCycleId (equal values). n8n consumers must migrate to the new topics + payload field and pin API version 2026-06-01 via the Replit-Version header before the legacy names are retired at Globus go-live (7/31). **Status as of 2026-09:** the dual-publish cutover has not completed — `season.*` events were still observed after 7/31 — so the Dispatcher keeps `season` as a legacy alias routed to the same Catalog Domain handler as `operational_cycle`, to be removed once Globus confirms the legacy topic is retired.

### Enriched application.created Payload

Subscribers no longer need to wait for later status_changed / readiness_changed events to backfill identity. Every application.created event now carries applicant, program, and season snapshots inline, plus a freshly-minted access token and redemption URL — so any subscriber (HubSpot, Mailchimp, partner n8n flow, custom webhook) can write the onboarding link back to the contact and send the welcome email regardless of which Hub path created the application. Each snapshot is best-effort: a failed lookup leaves that field null and the event still emits.

`applicationId / ... / applicantId / programId / seasonId / operationalCycleId / applicationType / enrollment / createdByStaffUserId / applicant / program / season / accessToken` 
## Integration Endpoint Contracts

The endpoints n8n and HubSpot transact against during the reference and activity loops. ESP (SendGrid) dispatch is owned by n8n's [Globus] Handler: Reference Domain workflow, not by the Hub — there is no inbound SendGrid webhook here; delivery acks come back via the dispatch-log callback below. Each endpoint is signature- or key-authenticated, idempotent on a stable provider identifier, and writes through the audit log. Endpoint implementation and MCP-registry registration are owned by the Globus core and integration runtimes; this table is the canonical docs-side contract.

| Method | Path | Purpose | Auth |
| --- | --- | --- | --- |
| POST | /api/reference-submissions/:id/dispatch-log | n8n delivery-ack callback. After the [Globus] Handler: Reference Domain workflow dispatches a referee email via the ESP, n8n posts back the dispatch outcome (timestamp, channel, recipient, providerMessageId, status) for the reference submission. The reference-channel specialization of the Dual-Callback dispatch-log. Idempotent on providerMessageId. There is no /admin prefix, and the Hub exposes no inbound ESP webhook for references — SendGrid dispatch is owned entirely by n8n. | n8n callback |

## Reporting Exports

Scheduled workflows generate reports and data exports from Globus data. These are typically sent to internal stakeholders or uploaded to shared drives.

Daily applicant pipeline summary / Weekly partner recruitment metrics / Monthly SEVIS compliance report / Season-end program analytics / Ad-hoc data exports for compliance audits
## Error Handling

> **Failure Isolation**
>
> When n8n workflows fail, the failure is isolated to the automation layer. Globus data is never corrupted by n8n failures. Failed workflows can be retried manually or automatically.

Failed workflows are logged with full execution context / Critical failures trigger internal Slack/email alerts / Automatic retries for temporary failures with increasing wait times / Unrecoverable failures are captured separately for manual review / Dashboard visibility for workflow health monitoring
## Webhook Contract Versioning

The public webhook contract is versioned by topic name, payload field name, and an HTTP version header. Two changes are currently in flight: the rename of the season.* topic family to operational_cycle.*, and the introduction of the Replit-Version header that gates response and payload shapes by date. This section is the canonical reference for both; every other docs page that touches a webhook payload field links back here.

### operational_cycle.* Topic & Payload Rename

Globus has renamed the four webhook topics in the season family to the canonical operational_cycle.* namespace, and renamed the corresponding payload field from seasonId to operationalCycleId. Note: this rename applies only to the public webhook contract — internal database columns (applications.season_id, rate-card scoping, RBAC user_scopes.seasonIds) and REST API query parameters are unchanged, and the Season business concept (Season records, Season pages, season-scoped filters) is unchanged.

| Legacy (deprecated) | Canonical |
| --- | --- |
| season.created | operational_cycle.created |
| season.updated | operational_cycle.updated |
| season.deleted | operational_cycle.deleted |
| season.status_changed | operational_cycle.status_changed |
| seasonId (payload field) | operationalCycleId (payload field) |

> **Dual-publish window**
>
> During the deprecation window, Globus emits both the legacy and canonical topic for every event and includes both seasonId and operationalCycleId in the payload. Consumers must cut over to the canonical topic and field before the window closes; after the cutover date the legacy topic and field are removed. New integrations should never bind to the legacy names.

### Replit-Version Header Gating

Public API responses and webhook payload shapes are gated by the date-based Replit-Version request header. Clients pin a specific date (currently Replit-Version: 2026-06-01) to opt into the canonical contract: operational_cycle.* topic names, operationalCycleId payload field, and any other forward-compatible shape changes. Requests without the header, or with an older pinned date, continue to receive the legacy contract until the deprecation window closes. This is why response shapes differ across versions and why the same event delivered to two consumers may carry different field names — each is reading its own pinned version.

## n8n Webhook Configuration

Webhook URLs for n8n integration are stored in the integration_configs table, making them database-backed and manageable through the admin UI at /admin/n8n-config. This eliminates hardcoded webhook URLs and allows per-brand configuration.

Webhook URLs stored in integration_configs table with key/value pairs scoped by brandId / Admin UI at /admin/n8n-config provides a visual editor for webhook URL management / Canonical setup uses the single unified webhook_url (all routing happens inside n8n); domain-specific keys (webhook_url_application, webhook_url_document) are only seeded from environment variables and are not consulted at delivery time (resolveWebhookUrl always returns the unified webhook_url) / Configuration changes take effect within 60 seconds due to the in-memory cache TTL / All configuration changes are audit-logged with the modifying staff member and timestamp

Permissions: `automations.manage` ("Manage automations and workflow integrations", restricted, no reason required) is seeded for System Administrator and deliberately denied to Hub Admin and Brand Admin, but no server route currently checks it. The n8n configuration routes under `/api/admin/n8n/config` are gated by `system_settings.manage`.
### Domain-Routed Webhook Resolution

The canonical pattern is the single unified Dispatcher (see Single Unified Dispatcher Endpoint ): only the general webhook_url is set, and n8n does all domain routing internally. Note: in the current code resolveWebhookUrl (server/webhook.ts) ignores the event type and always returns the unified webhook_url, so the domain-specific keys below are not consulted at delivery time (only webhook_url_application and webhook_url_document are even seeded, from environment variables).

| Event Type Prefix | Config Key | Fallback |
| --- | --- | --- |
| application.* | webhook_url_application | webhook_url |
| document.* | webhook_url_document | webhook_url |
| host.* | webhook_url_host | webhook_url |
| step.* | webhook_url_step | webhook_url |
| sevis.* | webhook_url_sevis | webhook_url |
| (any other) | webhook_url | N/A (general URL) |

### Configuration Cache

Webhook configuration is cached in-memory via integrationConfigCache.ts with a 60-second TTL. This prevents a database query on every event emission while ensuring configuration changes propagate within one minute.

### Delivery Health Dashboard

The admin delivery health dashboard is available at /admin/n8n-health and provides real-time visibility into event delivery status, retry rates, and dead-letter queue depth. Data is powered by the n8nDeliveryStats.ts service with entity name resolution for human-readable delivery records.

`GET /api/n8n/status` is a lightweight liveness probe for n8n: the handler applies no authentication and returns the standard success envelope with `status` (always `ok`), `service` (`globus-hub-erp`), `timestamp` and `uptime` in seconds. It does not call n8n or check webhook configuration.

## Integration Event Flows

Four named event flows move data between HubSpot, n8n, and Globus. n8n is the event bus in both directions — an extension of the "n8n is replaceable" sanctity rule above, not a contradiction of it. The producer and subscriber of every event remain HubSpot or Globus; n8n only transports. Payload field shapes referenced below are defined on HubSpot Integration .

## Webhook Configuration Matrix

Every named event/topic across the four flows above, with the system that owns the producer, the system that owns the subscription, and where n8n sits in the middle.

| Event / Topic | Producer | Subscriber | Routed via | Direction |
| --- | --- | --- | --- | --- |
| Application.pre_qual_status = qualified | HubSpot | Globus | n8n | Flow 1 — HubSpot → Globus |
| application.status_changed | Globus | HubSpot | n8n | Flow 2 — Globus → HubSpot |
| application.readiness_changed | Globus | HubSpot | n8n | Flow 2 — Globus → HubSpot |
| application.journey_changed | Globus | HubSpot | n8n | Flow 2 — Globus → HubSpot |
| notification.send_requested | Globus | HubSpot | n8n | Flow 3 — Globus → HubSpot |
| conversation.message_received (inbound) | HubSpot | Globus | n8n | Flow 4 — HubSpot → Globus |

The Flow 2 journey event is application.journey_changed (an earlier version of the routing matched application.journey_updated, which Globus never emits). Every application.* event reaches the Application Domain handler, which maps the real payload fields (newPosition / journeyPosition for journey, newState / readinessState for readiness) and includes a HubSpot property only when the event actually carries a value, so a field-sparse event can no longer blank a value an earlier event set. application.lifecycle_reconciled, application.submitted and application.stage_transitioned are tracked (last event type and time), but application.stage_transitioned and application.submitted are not yet written to a HubSpot stage or submission-time property because none exists. The handler's Prepare Application Data node reads the event's `status` field (which the Hub sets from the journey position) and its `statusMap` has keys only for `draft` (written as `created`), `submitted`, `in_review`, `matched`, `approved`, `rejected`, `dormant` and `closed`; any other non-empty value is written to `application_status` as `created`. The canonical journey positions (`application`, `validation`, `interview`, `placement`, `visa_sevis`, `pre_departure`, `in_program`, `completion`) are not among those keys. The handler's own code comment lists verifying this mapping against real values as an open item.

## Domain Action to n8n Flow

When a domain action occurs (e.g., a stage transition, document upload), the following pipeline delivers the event to n8n and triggers downstream side effects:

Domain action occurs (e.g., stage transition, document approval) / Service calls emitEvent() in eventEmitter.ts / Event persisted to ops_events table with status: pending / pushToN8n() called immediately (server/webhook.ts) / Payload enriched by domain: application events include applicant name, email, program, season; document events include document type and status; host events include org details and portal URLs / Webhook URL resolved from integration_configs table: the unified Dispatcher webhook_url is used (domain-specific override keys are not consulted) / POST to n8n webhook endpoint → ops_events status updated to completed / Side effects triggered: webhook dispatcher (to subscriber URLs with HMAC), queue rule evaluation (exception queues), reactive reconciliation (documents, SEVIS)
> **Events are persisted before delivery.**
>
> The ops_events table acts as a durable outbox. If n8n is unavailable, events remain in pending status and are retried. No events are lost, and delivery order is preserved per-entity.

## Workflow Version Control

All n8n workflows are source-controlled in the Alliance-Strategies/n8n-workflows GitHub repository. Workflows are treated as code artifacts — exported as JSON, committed to Git, and promoted through environments via pull requests and automated GitHub Actions.

### Branch Strategy

The repository's `development`/`qa`/`main` branches mirror three of the n8n instance topology's environments, plus a fourth `sandbox` branch for a standalone, disposable-testing instance with no CI gate or promotion path:

| Branch | n8n Instance | Purpose |
| --- | --- | --- |
| development | GKE, namespace `n8n-dev` on `globus-cluster-development`, publicly reachable at `development2.n8n.allianceabroad.com` / `development.n8n.allianceabroad.com` (confirmed 2026-09-16 — see the correction below) | Active development — free iteration, auto-exported (via PR) + auto-deployed on push |
| qa | qa.n8n.allianceabroad.com | QA validation |
| main | n8n.allianceabroad.com | Production |
| sandbox | GKE, namespace `n8n-sandbox` on `globus-cluster-development`, also reachable at sandbox.n8n.allianceabroad.com (public DNS/TLS now live) | Ad hoc workflow development / disposable testing, seeded from QA's curated workflow set. Has a manual-trigger (`workflow_dispatch`-only) integration-test harness but is not part of the automated promotion path and gates nothing |

**Sys tier (added 2026-09-23).** A fifth instance, `sys` (`sys.n8n.allianceabroad.com`, Cloud Run in `alliance-platform-services`), hosts the system/agent-fleet workflows (Security Agent, Overseer, GKE Health, Branch Drift, Asana Hygiene, PostHog, GitHub PR agents, IaC Sync, Vertex Sync, the daily Form-Engine Consumer Sync check, MCP monitors including the hourly Secrets Drift agent) so a bad business-side deploy cannot take the agent fleet's tooling down and vice versa. It is a real CI-deployable target (`deploy-sys.yml`) but is **not** part of the `development` → `qa` → `main` chain: there is no `sys` branch, workflows are authored on `development`, and each workflow migrates onto sys individually (24 agent and monitor workflows were moved off dev/qa, with their dev/qa id-map entries retired). `workflows/id-map.json` therefore has five environment columns — `dev`, `qa`, `prod`, `sandbox`, `sys`; a `sys` entry starts `null`, and the `__skip__` sentinel marks a slug as deliberately not applicable to an environment.

**Production topology.** Production n8n runs on GKE (namespace `n8n`, manifests under `k8s/prod/`): a single-replica Deployment of `n8nio/n8n` with a Cloud SQL Auth Proxy sidecar to the `n8n-db-prod` PostgreSQL instance, a persistent volume claim, a Service, and an Ingress for `n8n.allianceabroad.com` on a reserved static IP with a Google-managed certificate. (The committed `k8s/dev/` directory holds only an ingress manifest.)

> **Corrected 2026-09-16 — `development` DOES have public DNS/ingress after all.** The `no public DNS` claim below (accurate as of 2026-09-03) is now stale: a `kubectl get ingress -n n8n-dev` check confirms a live ingress for both `development2.n8n.allianceabroad.com` and `development.n8n.allianceabroad.com`, and both are externally reachable today (confirmed by directly curling n8n's webhook endpoints over the public internet, not just via port-forward). This matters concretely for third-party webhook configuration — e.g. SendGrid's Event Webhook must be pointed at `https://development2.n8n.allianceabroad.com/webhook/<path>` for dev, not at a port-forward-only address. See [Communications Administration](/docs/communications-administration#sendgrid-settings) for the exact per-environment SendGrid Event Webhook URLs and a real incident where a wrong path silently dropped every dev delivery event.
>
> **Corrected 2026-09-08 — production hostname.** Production is n8n.allianceabroad.com, not prod.n8n.allianceabroad.com as this page previously stated; verified directly against the live deploy-prod.yml Action configuration and the production ingress DNS/IP.
>
> **Corrected 2026-09-10 — `development` and `sandbox` instance identity, and Prod deploy status.** Two changes since the 2026-09-08 correction below, both verified directly against the live repo/cluster state, not asserted:
> 1. **`development` was repointed 2026-09-03** from the old public instance (`development.n8n.allianceabroad.com`) to an isolated GKE instance (namespace `n8n-dev`, cluster `globus-cluster-development`). `export-dev.yml`/`deploy-dev.yml` reach it via Workload Identity Federation + a `kubectl port-forward` tunnel opened for the duration of each CI job (CI does not rely on the public ingress), fetching the n8n API key from Secret Manager (`n8n-dev-deploy-api-key`) at runtime rather than a stored GitHub secret. The old public instance is separately slated for decommission — it was **not** renamed into `sandbox` (see next point).
> 2. **`sandbox` was replaced 2026-09-08** with a brand-new, isolated GKE instance (namespace `n8n-sandbox`, same `globus-cluster-development` cluster, seeded from QA's curated workflow set) — also not a rename of the old public dev instance. As of 2026-09-10 its TLS cert has finished provisioning and it's reachable at sandbox.n8n.allianceabroad.com over public DNS, in addition to `kubectl port-forward svc/n8n -n n8n-sandbox`.
>
> **Corrected 2026-09-10 — deploy-prod.yml has now run successfully against Production, and a `deployed-prod` tag exists.** The 2026-09-08 callout immediately below was accurate when written but is now stale: `deploy-prod.yml` completed two real, verified runs against Production on 2026-09-10 (33 canonical workflows updated on the first run; the second confirmed a clean, correctly-scoped no-op), and a `deployed-prod` git tag now exists and auto-advances on each successful run. Whether the separate `deploy-n8n-baseline.cjs` mechanism described below is still also in use has not been re-verified as part of this correction — confirm before assuming either mechanism is retired. Separately: the GitHub Environment's "optional manual approval" gate mentioned below is **not actually available** — required-reviewers protection rules were confirmed unsupported on this repo's current GitHub billing plan (2026-09-10 API check), so merges to `main` deploy immediately with no human checkpoint today.
>
> **Corrected 2026-09-08 — QA/Prod are not (yet) actually kept in sync via this repo's own deploy-qa.yml/deploy-prod.yml alone.** As of this writing, no `deployed-prod` git tag has ever been created in this repository — deploy-prod.yml has never completed a run against Production. QA and Prod's real, live workflow content is instead maintained via a separate, non-destructive pinned-source-commit deployer (`deploy-n8n-baseline.cjs`, in the `alliance-gcp-infra` repo, run via a Terraform `null_resource`), which creates only workflows missing from a pinned commit of this repo's `workflows/` directory and never deletes or force-overwrites an existing live workflow. Don't assume a merge to `qa` or `main` has already reached the live instances — confirm against the live instance (or the `deployed-qa`/`deployed-prod` git tag, once one exists) before relying on this branch's content as ground truth for what's actually running. **See the 2026-09-10 correction above: this is now out of date for Prod.**

### CI/CD Pipeline

> **Corrected 2026-09-16 — deploy-dev.yml now validates references BEFORE deploying, not only after.** A real incident (documented on the Dispatcher Routing Table section above) showed why the previous order was a gap: a post-deploy-only check can detect a broken commit but can't undo the live PUT that already happened. `deploy-dev.yml` now runs `validate_references.py` as a pre-flight step immediately after fetching the n8n API key, before `deploy_workflows.py` ever runs — a failure aborts the job before any content goes live. A second post-deploy run of the same check remains as cheap insurance.
>
> **Corrected 2026-09-10 — four Actions, not three, and export-dev.yml does not commit directly.** `development` carries a branch ruleset requiring every change to land via pull request (GH013), so `export-dev.yml` cannot and does not push straight to `development` — it pushes to a disposable `export-dev-workflows-<timestamp>` branch and opens a PR, left for a human to review/merge. There is also a fourth, previously undocumented Action, `deploy-dev.yml`, which deploys the *committed* `workflows/` content back onto the Dev instance (self-healing `id-map.json`, running the same reference/credential-binding validation gates QA and Prod get) — `development` is not purely export-only.

GitHub Actions automate the export and deployment lifecycle. All scripts use the n8n REST API (/api/v1/workflows) and require no native n8n Source Control license.

| Action File | Trigger | What It Does |
| --- | --- | --- |
| export-dev.yml | Push to development | Opens a WIF-authenticated `kubectl port-forward` to the Dev GKE instance, calls GET /api/v1/workflows, saves each workflow as an individual JSON file named id__slug.json, and — since `development` is a protected, PR-only branch — pushes to a disposable branch and opens a PR against `development` (not auto-merged) |
| deploy-dev.yml | `workflow_run` completion of export-dev.yml (i.e. after a PR merge to `development`), or manual dispatch | Validates cross-workflow references resolve on Dev (pre-flight, since 2026-09-16), then deploys the committed `workflows/` directory back onto the same Dev GKE instance via the id-map-aware `deploy_workflows.py`, re-runs reference validation (post-deploy insurance) and `validate_credential_bindings.py`, self-heals `id-map.json` (opening a PR with any updates), and advances a `deployed-dev` git tag on success |
| deploy-qa.yml | Merge to qa | Reads workflow JSONs from the qa branch and upserts them into the QA instance via PUT (update) or POST (create), resolving each workflow's target id through `workflows/id-map.json` first (see Persistent Workflow Identity in the n8n-workflows repo README) rather than trusting the committed file's own id directly |
| deploy-prod.yml | Merge to main | Same as deploy-qa.yml but targets the Production instance. References a GitHub Environment named `production`, intended to support an optional manual-approval gate — **confirmed 2026-09-10 that this gate is not actually available** on this repo's current GitHub billing plan (required-reviewers protection rules unsupported), so merges to `main` deploy immediately. Has now completed real, successful runs against Production (2026-09-10) — see the Branch Strategy corrections above. |
| ci.yml | Pull request into `development` that touches `workflows/**` (and manual dispatch) | Pre-merge validation gate — see Pre-Merge CI Checks below |
| backport-id-map.yml | Push to `qa` or `main` that changes `workflows/id-map.json` (and manual dispatch) | Runs `scripts/backport_id_map.py` to backport the pushed environment's own id-map column into `development`'s copy (only that column is filled on slugs development already has; a slug absent from `development` is added wholesale with its full row) and opens or updates one stably named PR (`backport-id-map-qa` / `backport-id-map-prod`) against `development`. A field that differs and is non-null on both sides is never auto-resolved: the run fails so a human decides. Exists because `id-map.json` is per-branch, so a self-heal on `qa`/`main` never reached `development` (incident 2026-09-21/22: `development` showed `qa: null` for a workflow live on QA for days) |
| bootstrap-prod-workflow.yml | Manual dispatch only; `only_files` required | The one sanctioned way to create a **new** workflow on Production (see Bootstrapping below) |
| deploy-sys.yml | Manual dispatch only; `only_files` required | Deploys the named workflow files from `development` onto the sys instance, validates cross-workflow references on sys, and opens an `id-map.json` update PR against `development`. Has no push trigger and does not advance a `deployed-sys` tag |
| export-sandbox.yml | Manual dispatch only | Exports the sandbox n8n instance (namespace `n8n-sandbox` on `globus-cluster-development`) through a WIF-authenticated `kubectl port-forward` and pushes the workflows straight to the `sandbox` branch (no PR). Sandbox is not part of the promotion path and the Action gates nothing |
| promote-sandbox-workflow.yml | Manual dispatch only; `workflow` required | Takes one sandbox-native workflow with no development, QA or production counterpart from the `sandbox` branch and opens a review PR against `development` proposing it as a new canonical file. It is a proposal, not an auto-merge |
| test-sandbox.yml | Manual dispatch only | Fires synthetic Globus events at the sandbox n8n instance and asserts the resulting executions succeed. Does not gate `development`, `qa` or `main` |
| notify-platform-pulse.yml | `workflow_run` completion of ci, Deploy to Dev, Deploy to QA, Deploy to Production, Export Dev Workflows or Test Sandbox | Posts to Platform Pulse through the gateway when a run on `development`, `qa` or `main` fails, and when a Deploy to Dev, Deploy to QA or Deploy to Production run on those branches succeeds |

### Pre-Merge CI Checks (ci.yml)

`ci.yml` is the repository's only pre-merge gate. It runs on pull requests into `development` that touch `workflows/**` (promotion PRs into `qa` and `main` rely on the deploy workflows' own validation instead). It has five jobs:

1. **Validate workflow JSON** — `scripts/validate_workflows.py`: JSON syntax and required n8n fields, scoped to the changed `workflows/*.json` files.
2. **Static deploy-safety checks** — `deploy_workflows.py --audit-ids-only` (no duplicate workflow names, no orphaned hardcoded workflow-id references), `validate_workflow_contracts.py` (the canonical finance-delivery contract) and `validate_credential_declarations.py` (a changed node that declares a credential-requiring auth mode with no binding at all, now including Webhook trigger nodes using header/basic/JWT auth). Added after a change merged clean and then broke every subsequent QA deploy.
3. **gitleaks secret scan** — scans the PR's commit range (`--redact`).
4. **Merge-conflict check** — polls the PR's mergeable state; a definitively conflicting PR skips the next job rather than running it.
5. **slopgate** (AI code-quality gate) — runs `slopgate --base origin/<base>` unless the PR is conflicting.

### Promotion-deploy validation

Beyond the table above, `deploy-qa.yml` and `deploy-prod.yml` each run `validate_workflow_contracts.py` before deploying and, after deploying, `validate_references.py` and `validate_credential_bindings.py --check-generic-credential-type` (Production allows only an unbound `netSuiteOAuth1Api`; `deploy-dev.yml` allows `hubspotAppToken`, `hubspotOAuth2Api` and `netSuiteOAuth1Api`). A `deployed-qa` / `deployed-prod` / `deployed-dev` tag advances only on a clean, full (no `only_files`) run, and later runs deploy only what changed since that tag. `deploy-prod.yml` also runs the WildPacks production email contract check and the orphaned-id audit, and sets `REQUIRE_EXISTING=true` so it refuses to create a workflow under a fresh id; `deploy-qa.yml` and `deploy-dev.yml` open an `id-map.json` PR after each run, `deploy-prod.yml` does not.

### Bootstrapping a New Workflow onto Production

Because of `REQUIRE_EXISTING=true`, a workflow whose `id-map.json` slug still has `"prod": null` can never go live through `deploy-prod.yml`. The **Bootstrap New Workflow to Prod** Action is the repeatable path: `workflow_dispatch` only, with `only_files` (comma-separated filenames) required. It first runs `scripts/audit_new_to_prod.py` and refuses if any named file already has a non-null prod id, then runs `deploy_workflows.py` without `REQUIRE_EXISTING` (scoped to exactly the named files), validates cross-workflow references on Production, and opens a PR against `main` recording the new prod id. The workflow always lands **inactive with no credentials bound**; a human must bind Production credentials to each node and activate it in the Production n8n UI before it does anything live.

### Required Repository Secrets

> **Corrected 2026-09-10 — Dev no longer uses a stored API-key secret.** Since the 2026-09-03 repoint to the GKE-based instance, `export-dev.yml`/`deploy-dev.yml` authenticate to GCP directly via Workload Identity Federation and fetch the n8n API key from Secret Manager (`n8n-dev-deploy-api-key`) at runtime — there is no `N8N_DEV_API_KEY` GitHub secret anymore.

The following secrets must be set under Settings → Secrets and variables → Actions in the repository before the Actions will run:

| Secret | Used By | Description |
| --- | --- | --- |
| GH_PAT | export-dev.yml, deploy-dev.yml | Personal Access Token with repo + workflow scopes. Used to push the export/id-map-update branches and open the resulting PRs against the protected `development` branch. |
| N8N_QA_API_KEY | deploy-qa.yml | API key for qa.n8n.allianceabroad.com |
| N8N_PROD_API_KEY | deploy-prod.yml, bootstrap-prod-workflow.yml | API key for n8n.allianceabroad.com |
| N8N_SYS_API_KEY | deploy-sys.yml | API key for sys.n8n.allianceabroad.com |

> **Credential portability across environments.**
>
> Workflow JSON files reference credentials by id+name, but that reference is **never trusted from the committed file** at deploy time. `deploy_workflows.py`'s `reconcile_credentials()` always resolves a node's credential binding from whatever is *currently* bound to that same node id on the deploy target, and strips the reference entirely for a brand-new node/workflow rather than pointing it at a nonexistent foreign id. In other words, promoting a workflow does not "bind by name" on the target — each environment's own live credential binding is authoritative and is preserved untouched by every deploy. Corrected 2026-09-10 after a related bug (a since-removed exception that let a production-only deploy mode trust the committed file's credential id) was found and fixed in `deploy_workflows.py`.

### Promotion Flow

Build and iterate freely in the Dev n8n instance as usual / The export Action runs on every push to development, opening a PR with any updated workflow JSONs (development is a protected, PR-only branch, so this is never a direct commit) / When a set of changes is ready for QA, open a Pull Request from development → qa and merge it / The deploy-qa Action fires on merge, upserting all workflows into the QA instance / After QA sign-off, open a Pull Request from qa → main and merge it / The deploy-prod Action fires on merge and, as of 2026-09-10, deploys immediately with no human approval checkpoint (see the Required Repository Secrets and Branch Strategy corrections above for why) — see the Branch Strategy callout above for the current, actually-in-use QA/Prod deployment mechanism
## Future Automations

As the ecosystem matures, additional automation workflows will be added:

Financial reconciliation and invoicing / Automated compliance scoring and risk flagging / Partner performance analytics and ranking / Predictive no-show modeling / Automated document verification (OCR + AI)