---
title: "Atlas: LLM-Guided Automated Testing"
slug: atlas-autonomous-testing
section: Governance & Operations
status: stable
description: "Atlas is an advisory testing agent."
last-updated: 2026-10-07
---

# Atlas: LLM-Guided Automated Testing

Atlas is an advisory testing agent. It reads Nexus, designs test scenarios with the Claude API, runs the confirmed ones in a real browser against the development environment, and reports to Platform Pulse. A person decides what happens to every result.

## Overview

Principle: agents recommend, deterministic checks prove, humans decide. The model writes text only and never executes anything. Every scenario passes a linter, every expectation cites a Nexus page, and a failure is a candidate until a person confirms it.

| | |
|---|---|
| Service | `atlas-agent`, Cloud Run (`us-central1`, project `alliance-platform-agents`), IAM-protected, development only |
| Scenario source | Kiwi TCMS, CONFIRMED cases only ([Test Case Standard](/docs/test-case-standard)). Three plans: "Atlas: regression" (BackOfficePrototype, 13 cases, daily and after each BOP dev-slot deploy), "Atlas: bop-front" (the staff front end, 28 runnable confirmed cases, daily and after each bop-front dev-slot deploy) and "Atlas: acceptance (applicant-update pilot)" (plan 53, 8 cases, never scheduled; see [the pilot contract](/docs/atlas-release-confidence-pilot)) |
| Model | Claude API, scrubbed Nexus documentation only; planning runs on a developer machine, the cloud service makes no model calls |
| Status of results | Advisory. Nothing Atlas produces blocks a merge, a deploy or a promotion |
| Code | Harness: the `Alliance-Strategies/atlas` repository (private, `main`, reviewed pull requests only; the only copy, since 2026-10-27). Service: `atlas-agent/` in the `vertex-agents` repository, which holds no harness code; `atlas-agent/ATLAS_REVISION.txt` pins the harness commit the service is built and tested from |

## What Atlas Never Does

- Run against production. Production hostnames are blocked as critical even if added to an allow-list by mistake.
- Run against QA today. The gate accepts QA hosts, but `atlas-agent` is registered for development only and QA has never been run.
- Block, approve, merge or promote anything.
- Confirm its own scenarios. Atlas can propose a case (it lands in Kiwi as PROPOSED); only a named person makes it CONFIRMED, and only a person can designate anything as blocking.
- Use a real person's account. Test identities are dedicated bot accounts; failed sign-ins are never retried. The one write the policy allows on shared development slots, the Globus sign-in form POST, is identity-bound: the posted email must name a catalogued bot (since 2026-10-28).
- Perform SEVIS or DS-2019 writes, payments, role or permission changes, credential resets, configuration publishing, or bulk operations. These are deny-listed in the harness.
- Send applicant, host or other personal data to the model.
- File, update or close tickets. A tracker adapter exists but no client is wired, so filing is refused.

## How a Run Works

| Stage | What happens |
|---|---|
| Fetch | The service reads the CONFIRMED cases of the Kiwi plan. If Kiwi is unreachable or nothing is runnable, the run does not start and Platform Pulse is told why. There is no bundled fallback |
| Lint | Each scenario is checked against the full scenario contract (fixed step vocabulary, an oracle from the deterministic registry, an executable assertion behind `ui_assert`, a value on every value-dependent JSON check) and the policy gate (allowed hosts, deny-list, identity). A CONFIRMED case that fails lint is not run: it is recorded in Kiwi as BLOCKED with the lint reason, never as IDLE (since 2026-10-27) |
| Run | Playwright drives a real browser as a bot identity; every request and redirect passes the gate; a failing scenario gets up to three attempts in total. A run pinned to a pool slot also asks that host for its own build identity (`GET /api/version`) at the start and the end of the run |
| Judge | Deterministic assertions decide the verdict: `pass`, `fail`, `flaky` (passed on a retry), `blocked` or `inconclusive`. The model is not the judge. A `verify_saved_state` step (since 2026-10-27) judges a write from an authoritative read-back of the record, never from the screen that reported success |
| Triage | A deterministic tree classifies a failure as product defect, scenario defect, environment or flaky. A scenario action that fails (a missing selector, a timeout) with no requirement-linked assertion behind it is a scenario defect (`stale_test`), routed to a person, never a product-defect candidate (since 2026-10-27) |
| Report | One Platform Pulse message (bot channel, HTML) and a Kiwi test run with one execution per case. `flaky` is recorded as ERROR, never as a pass. The Kiwi write is refused outright if the harness results do not match the materialized cases one to one (an unknown, missing or duplicate result id), so no status is ever inferred from absent evidence. A scenario in quarantine is listed separately and left out of the advisory verdict counts until a person releases it |

## How to Trigger It

| Mode | How | Notes |
|---|---|---|
| Manual | `POST /run-live` on the service | Needs a Google ID token and the API key. About 3 minutes for the 13-case regression set when it passes; retries on failing cases stretch it to 9-10 minutes |
| Daily | Cloud Scheduler jobs `atlas-agent-daily-regression` (06:25 UTC) and `atlas-agent-daily-bop-front` (06:50 UTC) call `POST /trigger/schedule` | Weekdays; each posts its own summary. Weekly jobs post the digest and the case-selection report on Mondays |
| After a deploy | `POST /trigger/deploy`, called by `atlas-after-deploy.yml` in BackOfficePrototype and by the bop-front deploy workflow once a feature branch has landed on a dev pool slot | Live since 2026-10-27: the caller is `atlas-ci-caller@alliance-platform-agents` (workload identity, repo and workflow pinned), the run targets the slot that was just deployed (`dev-N.globus.allianceabroad.com`), and the run record carries the deploy's commit and workflow run id. Several runs per evening are normal when several branches deploy (Kiwi runs 541 to 546 on 2026-10-27) |

Only one run executes at a time. Operator procedures (a failed run, a flaky scenario, a missing notification, a rollback) are in `RUNBOOKS.md` in the `atlas-agent` directory of `vertex-agents`.

## Where Evidence Lives

- **Kiwi TCMS**: the test run, one execution per case, with a comment on every non-pass carrying the verdict, attempts, failure class and reason. This is the durable record of pass and fail.
- **Platform Pulse**: the run summary with counts and the not-green scenarios, linking to the Kiwi run.
- **Sealed run record and screenshots**: written by the harness for every run. On a developer machine they are kept under `runs/`. On the cloud service every live run is also copied to a Cloud Storage bucket (`runs/<date>/<run_id>/record.json` plus an `evidence/` folder of screenshots, written once and never overwritten), so a failed run can be explained after the container recycles. Retrieve them with `GET /runs/{run_id}`, `/runs/{run_id}/evidence` and `/runs/{run_id}/evidence/{name}` (API key required); the run id is in the Platform Pulse message. Objects are deleted after 395 days, the recorded retention decision (2026-10-14). Runs from before 2026-10-07 were not stored; 455 records were stored for 2026-10-07 to 2026-10-27.
- **Build traceability** (since 2026-10-27): every run record carries `evidence` with the trigger-supplied identity (commit SHA, image digest, revision, workflow run id, Kiwi suite version), the harness commit that ran it (`harness_revision`, from `ATLAS_REVISION.txt` in the image), the tested host's own build as read from its `/api/version` at the start and the end of the run (`target.build`), the run's trigger as the service stated it (`manual`, `scheduled`, `deploy:<workflow run id>`), and an `assessment` that is `complete: true` only when every required field is present, the host answered, its build did not change during the run and it serves the claimed commit. Every shortfall is named in `assessment.reasons`; a missing or malformed value is recorded as absent, never inferred. The verdict is informational: it gates nothing.

## Guard Rails

- A host allow-list is the source of truth; anything not on it is refused, and production hostnames are refused even if listed.
- HTTP writes are denied by default. The deny-list names the only allowed writes (`writeAllowRules`); today that is the Globus sign-in form POST on development hosts, bound to a catalogued bot email. A write allowance for the Atlas sandbox slot (dev-6, applicant field edits only) exists as a draft pull request that only a named person merges.
- The deny-list, allow-list and per-run budget ceilings are policy files changed only by reviewed pull request.
- Prompts are scrubbed before they leave the machine, and a prompt containing a secret pattern is refused rather than sent.
- Bot credentials come from Secret Manager (cloud) or the environment (local), never from a scenario file.
- The harness subprocess gets a clean environment; the service API key is never passed to it.
- Atlas never weakens an assertion to make a scenario pass (healer guard).

## Model and Cost

The Claude API is the model provider, in a dedicated Anthropic workspace named `atlas`. A monthly cap of $150 is approved. The provider-side spend limit in the Anthropic Console is **not yet set**, and the budget alert and kill-switch drill are not yet built, so the cap is currently a policy statement rather than an enforced limit. Every model call is metered against a dated price table and recorded in a cost ledger; a typical planning run costs well under a dollar and total spend for October 2026 was $5.38.

## Autonomy

Decision rights are an earned ladder (policy P-001): rungs 0 to 4 are enabled by policy, but each rung's measured entry criteria are still enforced in code, so in practice only rung 0 (maintenance such as quarantining a flaky scenario) acts today. Rung 5, which covers blocking designations, the Product acceptance suite, budgets, the deny-list, egress, IAM, anything in production, and anything touching SEVIS or government systems, is human-only by design. Every autonomous decision is logged and revertible. The cloud service additionally ships its own unapproved copy of the ladder (`atlas-agent/autonomy-policy.cloud.json`, copied over the harness policy at image build), so no rung above 0 can act from the cloud whatever the harness repository approves.

The Phase 1 go/no-go was scored on 2026-10-27 against the criteria frozen in the plan (atlas `docs/GO-NO-GO.md`): flake rate, reproducibility, wall-clock and safety pass; held-out detection (3 of 7 injected defects), precision after triage and scenario validity fail. Result **no-go as frozen**; the recorded recommendation is to stay advisory at rung 0/1, with a named re-test per missed criterion. The decision rests with the owner.

## Ownership and Review

Owner: Platform Engineering. Changes to policy files, the deny-list, the price table or trigger authentication go through pull request review. Promotion of a scenario into the regression plan is a human CONFIRMED decision in Kiwi. An independent Engineering Team review of the original service pull request and its static-analysis suppressions is still outstanding; an AI review of the 22 machine-account-approved pull requests from 2026-10-08 was recorded on 2026-10-28 (Asana 1219349226136410) and its harness findings are fixed, but it does not replace the human read.

## Handover

What another engineer needs to run, deploy and roll back Atlas without the original owner. Recorded 2026-10-14 from `atlas-agent/RUNBOOKS.md` (written 2026-10-06); source-of-truth row updated 2026-10-27. Where this page and the runbook differ, the runbook's commands win, and anything it marks *verify* must be checked live. This section lists secret locations only, never values; never paste a key, token or password into a page, ticket or chat.

### Where everything lives

| What | Where |
|---|---|
| Service code, README, runbooks, security review | `atlas-agent/` in `vertex-agents` (`README.md`, `RUNBOOKS.md`, `SECURITY-REVIEW.md`) |
| Harness source of truth | The `Alliance-Strategies/atlas` repository on GitHub (private, branch `main`, pull request plus one approval, no force-push; established 2026-10-27, Asana 1219751768390748). It is the only copy: `vertex-agents` holds no harness code. `atlas-agent/ATLAS_REVISION.txt` there pins the harness commit; the deploy workflow checks that commit out into `atlas-agent/atlas/`, runs the harness's full test suite on it as a deploy gate, and builds the image from the same checkout. To ship a harness change: merge it in `Alliance-Strategies/atlas`, then open a one-line pull request in `vertex-agents` that changes the pinned SHA. Reproduce the harness without the owner's machine: clone the repository, check out the revision, `npm ci`, `npx playwright install --with-deps chromium`, `npm test` (the same steps both repositories' workflows run). The vendored copy that lived in `vertex-agents` until 2026-10-27 had drifted both ways; every difference was merged into the canonical repository before the copy was removed (Asana 1219920800593053) |
| Scenarios | Kiwi TCMS plans "Atlas: regression", "Atlas: bop-front" and "Atlas: acceptance (applicant-update pilot)" (Kiwi runs in `alliance-platform-services`) |
| Cloud Run service | `atlas-agent`, `us-central1`, project `alliance-platform-agents`. The service itself is not in Terraform (live only) |
| Daily schedule | Cloud Scheduler jobs `atlas-agent-daily-regression` and `atlas-agent-daily-bop-front` (weekdays), `atlas-agent-weekly-digest` and `atlas-agent-weekly-select-report` (Mondays), service account `atlas-scheduler`, declared in `alliance-gcp-infra` |
| API key and bot credentials | Secret Manager in `alliance-platform-agents`: `atlas-agent-api-key` and three bot-credential secrets (names in `cloudbuild.yaml`); declared in `alliance-gcp-infra` `environments/vertex`; originals in 1Password (Globus Dev vault, items named "Atlas - ...") |
| Gateway PAT (posts to Platform Pulse) | Secret `atlas-agent-gateway-pat` in the dev project `skilled-script-469819-f2`, with `secretAccessor` for `atlas-agent-sa`. It belongs to the gateway robot `atlas-agent` and **expires**; rotate through the gateway admin portal, then add a new secret version |
| Run evidence | Kiwi test runs, plus the Cloud Storage run store described above |
| Atlas sandbox slot | `globus-dev-6` (dev-6.globus.allianceabroad.com), reserved for Atlas by namespace labels (ticket 1219650054770873, 7-day renewable hold since 2026-10-28). The Atlas staff bot `atlas-e2e-staff` holds `system_admin` in every dev pool slot database. A seed/reset job and the write allowance are still to be delivered |

### Deploy order

1. Confirm every secret above exists with a value. A missing value fails the deploy on purpose.
2. Build the image with `cloudbuild.yaml` under a **non-`latest` tag** (for example `:v0.4.0-<sha>`). `:latest` is not trusted.
3. Deploy that image by hand under a human identity, using the same flags as `cloudbuild.yaml`. The pipeline's own deploy step currently fails because the build service account lacks `iam.serviceaccounts.actAs` on `atlas-agent-sa`. When `--set-env-vars` holds a comma-separated value (for example several emails in `ATLAS_TRIGGER_CALLERS`), use the `^|^` delimiter.
4. Verify: `GET /health` reports the expected mode; `POST /run-live` without the key returns 401; one manual run shows `policy_events` of 0, verdict counts matching the last good run, and a Platform Pulse message arriving.

### Roll back

Revisions are immutable, so a rollback is a traffic move, not a rebuild.

1. List revisions: `gcloud run revisions list --service atlas-agent --region us-central1 --project alliance-platform-agents`
2. Move traffic: `gcloud run services update-traffic atlas-agent --to-revisions <good-revision>=100 --region us-central1 --project alliance-platform-agents`
3. Verify as in Deploy order step 4.

Rolling back past the Kiwi-sourced release (v0.4.0) returns to revisions that bundle their own scenarios, so results stop being written to Kiwi.

### Stop it

Pause the schedule with `gcloud scheduler jobs pause atlas-agent-daily-regression --location <job location> --project alliance-platform-agents` (check the location with `gcloud scheduler jobs list`) and resume with `jobs resume`; do the same for `atlas-agent-daily-bop-front`. The budget kill switch is not built yet, but the cloud service makes no model calls, so cloud spend is Cloud Run time only.

## Roadmap

Recorded 2026-10-14 from every ticket on the Asana board "Automated Quality & Release Reliability" (70 tickets, including closed: 64 done, 6 open) and the post-v1 list in the Atlas plan; status column refreshed 2026-10-28 from the ATLAS board. Items under Later are **not committed**; the owners shown are proposals, not assignments.

![Atlas roadmap: current state, next open tickets, and later post-v1 items that are not committed](/atlas-roadmap.svg)

### Current state

Development only, advisory results, a person decides.

| Area | What exists |
|---|---|
| Foundation | Plan reconciled with decisions taken, harness repo and policy files, Nexus corpus ingest, Claude-API planner, portal discovery (35 staff pages, 864 controls), scenario pipeline with cost ledger |
| Service | `atlas-agent` v0.4 on Cloud Run, Kiwi TCMS as the only scenario source, durable run store, Platform Pulse reporting, network egress allow-list, budget alert and kill switch, runbooks, security review |
| Loop | Promotion through Kiwi, evidence stamping, RBAC matrix suite, flake and run-time fix, autonomy ladder (policy P-001), weekly digest, daily regression on a development pool slot, after-deploy runs from BackOfficePrototype and bop-front |
| Quality-controls pilot | POC 01 to 07 and the Agentic CI/CD discovery, all closed |
| Evidence so far | Two real product defects found by scenarios: the Compliance Console 400 (Asana 1219148024372367) and the matching weight panel refusing seed-normalised profiles (Asana 1219661499926252). Two data points, not a detection rate |

### Next: open tickets

| Ticket | What it delivers | Status 2026-10-28 |
|---|---|---|
| 1.7 Real-app replay scoring | Score the replay set against the real applications, not fault-injection stand-ins | Done: 8 injected bop-front defects, 3 of 7 caught before detectors were added |
| 1.8 Scored go/no-go report | Replaces the provisional evaluation memo | Report merged (no-go as frozen); the retrieval question-set review by a person is still open |
| 3.4 PR advisory check | Advisory comment on merge candidates and release slots | Advisory case selection shipped; auto-skip stays off |
| 3.5 QA support and first QA run | Dev-vs-QA differential scenarios; QA has never been run | Deferred by the owner |
| 4.1 Metrics, cost per defect, recommendation | 30-day analysis and recommendation memo; decides the path in the decision gate below | Window closes 2026-11-06 |
| Release-confidence pilot RC-01 to RC-09 | One approved applicant-update outcome proven on a sandbox slot, then advisory QA evidence, then opt-in gating only after a named approval | Contract drafted (delegated adoption, not approved), verifier shipped, Kiwi plan 53 created, sandbox slot dev-6 reserved; write allowance and reset job pending a named person |

### Later: post-v1, not committed

| Item | Proposed owner |
|---|---|
| Coverage of Teach USA and the remaining programs, Partner Portal, WildPacks WordPress flows | Sam, with a Product triage partner named by Bryan |
| Doc-drift auto-proposals to the docs-update flow (the agent proposes, a human merges) | Sam |
| Visual regression and accessibility probes | Sam |
| Governed, budgeted load testing, and a synthetic-data generator | Sam, with Bryan for budget |
| Authorisation fuzzing (security-sensitive, heavy governance) | Sam, with a named security reviewer |
| "Ask Atlas" read-only assistant, and release-note and acceptance-criteria coverage generation | Platform Engineering, with Product/Quality (Mubashir) |
| Fine-tuned or distilled small models, and cross-environment parity checks bridging into the config-parity drive | Sam |

### Decision gate

The 4.1 recommendation chooses between staying advisory, partial gating, or stopping; the proposed decider is Bryan. The Later items are not committed and depend on that outcome. Blocking designations, production, SEVIS and government systems stay human-only. Optional read-only production monitoring (plan question Q17) is decided after this gate, with security sign-off.

## Known Limits

- Atlas has found two real product defects with scenarios (a Compliance Console API contract mismatch; a matching-weight panel that refuses every seed-normalised profile). Two data points, not a measured defect-detection rate. On 8 injected bop-front defects the library as it stood caught 3 of 7 non-equivalent ones; detectors now exist for the rest, but that is no longer held-out.
- Triage precision was low before the `stale_test` class: 166 `product_defect` labels in the local run records held 2 real defects. Expect a re-score after 30 daily runs with the new triage.
- Several Globus staff page-load scenarios are flaky, and their retries are what stretch a run from about 3 minutes to 9-10 minutes. `S-J01-3004` (a text-absence check racing page hydration) fails on every slot and is the usual one non-green row.
- Configuration and infrastructure drift (Terraform, secrets, CORS, workload identity) is invisible to browser scenarios.
- A client that stops sending its session cookie is invisible to browser scenarios: the harness request gate re-issues every page request through Playwright with the browser's cookie jar, whatever credentials mode the page asked for. That class of regression is covered by a build-artefact check in the bop-front contract suite, not by a scenario (recorded 2026-10-27).
- Write scenarios that were confirmed before the sandbox existed (for example the portal-base-URL rejection cases) run on shared dev slots and are blocked by the request gate as designed; they belong in the sandbox set.
- Development only: QA and production behaviour are not tested.

## Related Pages

- [Test Case Standard (Kiwi TCMS)](/docs/test-case-standard): how cases are written and logged
- [Testing Practices](/docs/onboarding-testing-practices): what "tested" means for a code change
- [Vertex Agents](/docs/vertex-agents): the agent fleet and registry
- [Atlas Release-Confidence Pilot: Acceptance Contract](/docs/atlas-release-confidence-pilot): the one synthetic applicant-update outcome the pilot must prove, with the draft runbook, coverage table and gap register (draft, adopted by delegation, awaiting Product/Quality approval)