Skip to content

Testing

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

Every test layer and tier in this repository: what it proves, how to run it, whether CI runs it for you, and where its results are recorded. This is a map of the tiers, not an index of individual test functions — there are thousands of those across hundreds of test modules, and one added inside a suite that already runs needs no change here. make test-list enumerates the suites themselves, and the section on what make test does not run is the one place the suite-level exceptions are written down. If you are looking for what a given release was actually validated against, that is the Release Validation Records; this page describes the methods, not any particular run of them.

The organising fact: most of what protects this repo runs on every pull request, but the tiers that need a deployed stack cannot. Roughly a dozen methods below run only when a person asks for them, which is why each one has a written procedure and a record of its last run.

LayerNeeds AWS?Runs in CI?Entry point
1. Offline test suitesno✅ both CIs, but through two other targets — see belowmake test
2. Static gatesno✅ both CIsmake lint-cicd · make typecheck · make terraform-check
3. Web UI unit testsno✅ both CIsmake ui-test
4. Security scanningno✅ both CIsmake srt-scan · make dep-audit
5. Integration smoke suiteyes⚠️ GitLab scheduled (primary suite nightly) + manual button — not a merge gatepipeline integration_tests
6. Live-stack tiersyes❌ manual, except the ZAP DAST and deploy-variant rows (GitLab weekly schedule, while it is active)see the table
7. Benchmarksyes❌ manualmake benchmark-release

make test and make lint are the two commands to run before opening a pull request. make all is both.

make test discovers and runs every non-integration suite in the repo — the idp_common library, idp_cli, idp_sdk, idp_feature_sdk, the feature platform, the per-Lambda suites, the config library, and the repo’s own tooling tests under scripts/, with the handful of documented exceptions below. make test-list prints the discovered roots without running them, which is the honest answer to “is my new suite actually being run?”.

Terminal window
make test # everything, auto-discovered
make test-list # which roots were discovered
cd lib/idp_common_pkg && make test-unit # one root, isolated
make test-integration-all # only the integration-marked tests (needs AWS)

make test itself runs in neither CI. CI runs the same suites through three other targets — make test-cicd -C lib/idp_common_pkg, make coverage-all-cicd, which runs every suite a coverage tree measures, and make test-packages-cicd for the rest — and the last of those is a hand-enumerated recipe, so a suite that make test discovers but nobody adds to it runs locally and on no pull request. 22 registered roots holding 506 tests were in exactly that state, including the resolver suites covering the download allow-list and the discovery upload target. scripts/tests/test_src_lambda_tests_in_ci.py now closes that by deriving both sides — every directory holding a tracked test_*.py, against the paths the two recipes run pytest against plus the directories the coverage trees measure — so a root that is in neither is a test failure rather than a gap somebody finds by hand months later. A suite that genuinely cannot run in the shared gate goes in the excluded registry below with a reason.

Both of those targets run every suite with the AWS environment removed — no region, no credentials, no profile, the shared AWS config file neutralised and the instance metadata service disabled — because that is what a CI runner supplies. The wrapper is defined once, in make/hermetic_aws.mk, and included by the root Makefile and by lib/idp_common_pkg/Makefile; make test-integration there deliberately does not use it, because integration tests need real credentials.

A suite that needs a region or placeholder credentials must therefore supply them in its own conftest.py, where os.environ.setdefault reinstates them after the wrapper has taken the machine’s real ones away. Inheriting either from the shell means the suite passes where it was written and fails where it is gated, and there are two distinct ways that happens:

  • A region. A Lambda handler the suite imports builds a boto3 client at module scope, so importing it needs a resolvable region and botocore raises NoRegionError during collection without one. A developer machine supplies one from the shared AWS config file; a runner supplies nothing.
  • Credentials. botocore freezes the session’s credentials object into a client when the client is constructed, so a client built at import time with none resolvable can never sign — no matter how many credentials appear later, including the ones moto sets when a fixture starts. A developer box on EC2 resolves credentials from the instance metadata service without anyone noticing; a runner cannot reach it. The symptom is AttributeError: 'NoneType' object has no attribute 'access_key' out of botocore/auth.py, which does not mention credentials at all.

A corollary worth knowing: anything a test module does to os.environ at import time affects the whole session. -m "not integration" deselects tests after collection has imported their modules, so even a file whose tests never run can change the environment every later module is imported under.

scripts/tests/test_offline_suites_are_hermetic.py checks all of this, and checks the stripping wrapper itself by measurement: it builds an environment that supplies a region and credentials through every source, puts it through the wrapper, and requires that neither survives.

There is no standing failure set — Expected standing failures: 0 on a correctly installed tree, and the enumerated list of accepted failures in full-test-battery is empty. So treat any failure as a real regression until proven otherwise. scripts/tests/test_standing_failure_baseline.py holds that claim, this page and the two skills that repeat it to the same number, so they cannot drift apart again, and make test itself compares the failures it observed against that table — a failure it does not declare and a declared row whose test passed both fail the run.

Most surprising failures are still a stale virtualenv missing the pinned [test] extras — but do not expect a broken install to announce itself as an ImportError. Several Lambdas catch a missing idp_common on purpose and degrade (feature-platform/main-stack-extensions/lambdas/apply_feature_config_preset/index.py logs at ERROR and applies a config preset without recording a revision), so a suite that exercises the non-degraded path fails on a bare assertion instead. Two tests in test_apply_feature_config_preset.py (test_remove_hands_the_pipeline_back_to_default_then_deletes and test_remove_keeps_an_active_profile_when_there_is_no_default_to_fall_back_to) were misread as a standing failure of this repo for exactly that reason. With idp_common unimportable that file reports 2 failed, 18 passed; with the pin the make targets export (PYTHONPATH naming every lib/* package root, absolute) it reports 20 passed, on the same interpreter and the same commit. Measured identically under Python 3.12 and 3.13, so it is not a version incompatibility. Check that python3 -c "import idp_common; print(idp_common.__file__)" resolves inside your own checkout before reading anything else — an editable install can silently point at a different, or deleted, checkout. The diagnosis order, the per-suite expected totals, and how to prove a failure is inherited are in the full-test-battery procedure. Conventions for writing tests — pytest markers, moto, conftest layout — are in testing-qa.

Auto-discovery means a suite cannot be forgotten, not that every suite is run. scripts/run_all_tests.py refuses to run at all when it finds a directory holding a test_*.py that is in neither of its two registries — the roots it runs, and the roots it excludes with a written reason — so tests in a new location cannot be silently skipped. An exclusion covers only the directory named: nesting under an excluded directory used to inherit the exclusion, which meant excluding scripts quietly accepted every future test directory beneath it, so each excluded directory is now listed on its own. That check backs make test, which runs in neither CI, so scripts/tests/test_testing_doc.py re-derives it on every pull request, where pytest scripts/tests does run.

These are the directories in the excluded registry. They are listed because a suite that exists and never runs is otherwise indistinguishable from one that passes:

Not run by make testWhy
scriptsscripts/test_api_rbac.py is the live RBAC harness driven by make api-test against a deployed stack (layer 6), not a pytest suite; collecting it picks up its test_email() helper as a test
src/lambda/ocr_benchmark_deployertest_local.py needs huggingface_hub, which is not a test dependency
samples/lambda-hook-inference/GENAIIDP-chandra-ocr-hooktest_local.py is a manual local-run script and collects zero pytest tests (measured)
lib/idp_sdk/idp_sdk/_coresource, not tests: test_studio_processor.py is the Test Studio processor module, which the test_ prefix makes look like a suite
lib/idp_common_pkg/manual_tests/agentsoperator-run scripts, not a suite: each one drives real Bedrock, Athena or DynamoDB against a deployed stack and bills model calls. Run by hand (python manual_tests/agents/test_analytics.py -q "…"); norecursedirs in lib/idp_common_pkg/pytest.ini keeps a bare pytest from collecting them

Adding an exclusion, or lifting one of these, fails that guard until this table and the registry agree — it is checked in both directions, so a row that outlives the exclusion it describes fails too.

Separate from the directory exclusions above, the tests marked @pytest.mark.integration are deselected from every suite on this page. The mechanism is the runner, not a config file. scripts/run_all_tests.py runs one pytest -m "not integration" <root> subprocess per root, so the filter applies to every root it discovers — including a new package under lib/, which inherits nothing. Only lib/idp_common_pkg/pytest.ini carries addopts = -m "not integration" of its own; lib/idp_sdk and lib/idp_cli_pkg have an empty addopts, and the root pytest.ini deliberately sets none, because an addopts there would silently change collection for every suite that resolves to it.

Those tests call live AWS — real Bedrock, Textract, S3 and DynamoDB rather than moto — and within idp_common they are the only ones that do. (Layers 5 to 7 below need AWS too, but none of them is a marked pytest suite.) They run with make test-integration-all, or per-package with make test-integration, and nothing else runs them: not make test, not GitHub, not GitLab. A person typing the target is the whole of their coverage.

⚠️ GitLab’s integration_tests job is a different thing with a similar name. It is the deploy-driven smoke suite in layer 5 — a CodePipeline stack deployment — and it never invokes pytest -m integration. Reading the stage name as coverage of this tier is what let 14 of its 23 tests sit failing on tooling faults until a release validation found them (#1307): the tier was uncovered, and the exemption recording that it was local-only cited the stage as the reason it did not need to be. test_ci_gate_parity.py’s test_the_integration_pytest_tier_runs_in_no_ci now measures the absence instead of asserting the stage name, so if a CI starts running the tier, the stale exemption fails rather than the claim going quietly out of date.

Two consequences worth keeping in mind when adding one of these tests. The marker, not the directory, is what makes a test part of this tier — make test-integration runs pytest -m "integration" over its whole package tree, so an integration-marked test under tests/unit/ is in it. And because nothing gates them, a broken one stays broken: a test whose seams are all mocked needs no credentials and belongs in the default gate, where something will notice.

idp_common and the SDKs are editable installs, so import idp_common reads whatever pointer is in the active interpreter’s site-packages — not necessarily this checkout. Where more than one checkout is worked on at once and python3 resolves to a shared interpreter rather than a per-project virtualenv, that directory is shared and the last pip install -e wins for all of them. make test-cicd is itself a writer: its test-unit-cicd target runs pip install -e ".[test]" unless SKIP_INSTALL=1 is set, so running the gate repoints the pointer as a side effect.

Nothing raises when this happens. The imported package is real and self-consistent, just a different revision, so it shows up as an unrelated-looking assertion failure or as a green run whose coverage number describes another tree.

Through make, the pin is applied for you. PYTEST_HERMETIC in make/hermetic_aws.mk — the wrapper every pytest invocation in both Makefiles goes through — exports an absolute PYTHONPATH naming every first-party root of the checkout the makefile belongs to, and scripts/run_all_tests.py passes the same value to each of the pytest subprocesses make test starts. The set is derived from lib/*/pyproject.toml, the same rule that decides what FIRST_PARTY_EDITABLES installs, so a package added under lib/ is pinned without anyone adding it anywhere; scripts/tests/test_first_party_pythonpath.py asserts the three implementations of that rule agree, and measures the wrapper by importing all five packages under it. A caller’s own PYTHONPATH is kept, after the pin. make test FIRST_PARTY_PYTHONPATH= suppresses it, for deliberately testing an installed copy.

Running pytest directly, pin it yourself — every root, and absolute. The packages import each other, so a pin naming lib/idp_common_pkg alone is refused by the next one; and a relative pin is dropped by any subprocess started in another directory, which several suites do start. scripts/tests/first_party_provenance.py is what refuses, and it is called from lib/idp_common_pkg/tests/conftest.py, feature-platform/main-stack-extensions/tests/conftest.py, scripts/tests/conftest.py, scripts/tests/test_model_surface_consistency.py and lib/idp_sdk/tests/unit/test_config_operations_region.py. It compares checkout identity, deriving each side’s root by walking up to .git, so a git worktree validates against itself and passes while a worktree nested inside another checkout is correctly refused. Before refusing it tries to repair: it puts this checkout’s own roots first on sys.path and reports having done so, since that fixes the process in hand and not the environment. It still refuses when the package was already imported — re-importing one other modules hold references to leaves two live copies — and when this checkout has no copy to prefer. A refusal is not a failing test; it is a run that did not happen, and the message says so, because a collection error at the end of a long log reads as “some tests failed”.

Fix it durably by installing with the interpreter you actually want (<your-venv>/bin/python -m pip install -e "lib/idp_common_pkg[test]", by path and never by bare name — see dependency-confusion.md). scripts/check_first_party_deps.py, which runs in both CIs and after every make setup, reports an editable pointer into another checkout as a failure — it answers “from source, and from which source”, where answering only the first half is how this condition stayed invisible while the control that looked like it covered it was green. IDP_ALLOW_FOREIGN_FIRST_PARTY=1 downgrades both that failure and the pytest-side refusal to a note when you are deliberately testing an installed copy; it is a per-invocation switch, so it is not in scripts/tests/gate_exemptions.json, for the reason that file records for ALLOW_SHARED_BRANCH.

2. Static gates (lint, types, and hand-written scanners)

Section titled “2. Static gates (lint, types, and hand-written scanners)”

make lint is the local gate; make lint-cicd is the same set in check-only mode and is what both CIs run. Several members of the set are not linters at all but hand-written scanners for classes of defect that have each shipped at least once:

GateWhat it catches
make ruff-lint · make formatPython style and formatting (88 cols, Python 3.12) over every tracked .py file except a named per-file debt list
make check-lint-debtthat debt list staying honest: an excluded file that gained a finding, one that is now clean and should be delisted, a dead path, a bare directory name in one of the three generated arrays, or a tracked file ruff’s walk never reaches (which is what covers the top-level exclude array, whose bare names are deliberate). ruff.toml excluded five bare directory names until #975, which match at any depth, so 442 of 1230 tracked .py files were read by neither gate. Use python3 scripts/check_lint_debt.py --explain <path> to ask whether one file is linted — no ruff invocation answers that correctly
make check-ui-i18n-catalogthe UI’s committed English message catalog (src/ui/src/i18n/en.json) drifting from the defineMessages calls it was extracted from, and a translation catalog that would fail the UI build (malformed ICU, an argument or plural the English does not have, an id the English no longer has). Check-only: it re-extracts into a temporary file and diffs; make ui-i18n-extract is the writer. Without src/ui/node_modules, or on a machine the installed @formatjs/cli ships no native binary for, it reports a skip, never a pass; in CI either is a failure (#1770)
make check-ui-string-debta hardcoded user-facing string added to the UI: a file carrying more findings than scripts/ui_string_debt.json pins for it, a file not in that baseline carrying any (a new file and a converted file both start at zero), an entry that is now vacuous or names a vanished path, or a counting configuration (src/ui/eslint.i18n.config.js) whose hash is not the one the counts were measured with, so a widened exclude list cannot lower every count silently. Strings inside const ALL_CAPS = ... declarations are not counted (a declared gap, with its numbers, in the ratchet’s docstring). Use python3 scripts/check_ui_string_debt.py --explain <path> to see one file’s findings; --write re-records a smaller count and refuses a larger one without --allow-new-debt "<reason>" (#1770)
make cfn-lintany file declaring AWSTemplateFormatVersion — discovered by content, so a new template cannot escape it. Fails on errors only
make check-arn-partitionshardcoded arn:aws: / amazonaws.com instead of ${AWS::Partition} / ${AWS::URLSuffix} — GovCloud compatibility
make check-filtered-scansDynamoDB Scan with a filter expression that cannot see all matches
make check-data-plane-tagsthe idp:plane=data tag on the Lambdas that must carry it
make check-markdown-linksa relative Markdown link that resolves to nothing, an #anchor naming a heading that has since been retitled, a published docs/ page linking relatively to one docs-site/setup.sh does not symlink (resolves on GitHub, 404s on the site), and a code fence that swallows content, which would otherwise leave this gate blind to the rest of the file. Every tracked .md file, from git ls-files. Anchors are slugified by github-slugger’s rule, which scripts/tests/test_markdown_links.py verifies against the real package over every heading in the tree. External http(s) URLs are not fetched, so a green result says nothing about them (#1068)
make validate-buildspecmalformed CodeBuild buildspecs, and a loop in a multi-line command that cannot report its own failure — otherwise a deploy-time failure
make codegen-checkgenerated GraphQL types drifting from the schema
make typecheckbasedpyright over every tracked .py file — this is what both CIs run. make typecheck-pr narrows it to the files a branch changes, for local latency, and is a gate in neither
make api-test-statican API operation added without authorization, and drift between the dispatcher’s generated required-groups manifest and scripts/api_rbac_expectations.yaml — see layer 6 for the live half
python3 scripts/check_first_party_deps.pya first-party package in the current environment that came from a package index rather than from lib/, which on public PyPI is somebody else’s code
scripts/tests/test_doc_install_commands.py (part of make coverage-all-cicd)a pip install documented in a fenced code block that could resolve a first-party name from an index — a bare name, or a path install missing a sibling the package requires by name. The environment checker above cannot see an instruction nobody has run yet
python3 scripts/sdlc/validate_service_role_permissions.pythe CloudFormation service role missing a permission the templates need
scripts/tests/test_agent_capabilities.py (part of make test-packages-cicd)an agent, tool, MCP tool or claude -p harness with no entry in scripts/tests/agent_capabilities.yaml, an entry whose surface is gone, and an entry that takes document-derived or external text and reads private data with an external output that no named control limits and no recorded finding lists. Surfaces are derived from source by AST. It checks the manifest’s declarations, not whether they are true; the agent-capability-review skill is the audit that re-reads them. scripts/tests/test_analytics_role_grants.py bounds the IAM grants of the roles those entries name
scripts/tests/test_threat_model_evidence.py (part of make test-packages-cicd)an entry under security/threat-modeling/ (every one the export builder reads) whose mitigations neither cite a test, gate or make target that resolves nor say UNVERIFIED, and a rise in the pinned count of UNVERIFIED entries

Three of these gates guard the gates themselves: scripts/tests/test_ci_gate_parity.py fails if a gate runs in one CI and not the other, or if lint-cicd becomes weaker than local make lint; scripts/tests/test_nested_stack_parameters.py checks parent-to-nested stack parameter wiring that cfn-lint’s own rule cannot see; and scripts/tests/test_lint_debt_gate.py drives check-lint-debt through each failure mode it claims, because a ratchet nobody has watched fail is not a ratchet. All three exist because every gap they cover was originally found by hand, months late.

make typecheck reads every tracked .py file — git ls-files '*.py' | wc -l and basedpyright’s filesAnalyzed agree exactly, and that identity, rather than any particular count, is what scripts/tests/test_pyright_config.py asserts by deriving the set from git ls-files rather than from a list. Run those two commands for today’s figure rather than looking for one on this page: it changes with almost every merge, and a written-down count has gone stale in three separate documents at once. The include array once named six paths and reached 432 of the 1,230 tracked at the time, and two NameError-class defects reached develop through the gap.

Reading every file is a weaker property than it sounds, and the same suite now covers the difference. basedpyright honours PYTHONPATH, which neither the make target nor either CI sets, so with reportMissingImports configured "none" the shared idp_common library did not resolve and no call into it could produce a diagnostic — zero errors over a file count that matched git ls-files exactly, hiding eleven real ones (#1109). pyrightconfig.json’s extraPaths fixes resolution in the configuration rather than the environment, using relative paths so it cannot resolve against another checkout, and the suite asserts that first-party imports really do resolve — by running basedpyright, not by inspecting the JSON.

make terraform-check is a static gate that is not part of make lint, because it needs network access and about two gigabytes of cache. Each CI runs it as its own job, terraform_checks. It works in four steps:

  1. It translates template.yaml, with the nested templates read from source, into a temporary directory using terraform/translator in --offline mode. No AWS call is made.
  2. It installs each binary in TERRAFORM_CHECK_BINARIES into a cache. By default these are the floor and latest-stable releases of both Terraform and OpenTofu, pinned in the Makefile and mirrored in both CI configs. An archive is accepted only when its SHA256 matches both the release’s published SHA256SUMS and the hash committed in scripts/terraform_binaries.lock.json.
  3. It runs init -backend=false and validate on the generated module with each binary. Provider versions are pinned exactly through an override file: aws and time from the lock file, and awscc to the vendored schema’s version. Errors fail the gate. Warnings are counted and printed but do not fail it.
  4. It runs make terraform-scan-diff: a pinned checkov (CHECKOV_VERSION in the Makefile) over the CloudFormation templates and over the generated module, failing on a Terraform finding the CloudFormation side does not have unless scripts/terraform_scan_baseline.json explains it. Findings on checks with no CloudFormation counterpart are pinned per check and resource type in scripts/terraform_scan_tf_only_baseline.json, so a new one fails too.

The gate proves that the generated HCL parses, that its references resolve, and that every resource and attribute type-checks against the pinned provider schemas, on the oldest and newest releases the module claims to support. It does not plan or deploy anything, so a value AWS would reject at apply time, an IAM policy grammar error, or an ordering problem between resources is outside what it can see. The translator’s and bridge’s own unit tests run in make test (layer 1) and are not repeated here. scripts/tests/test_terraform_check.py covers the gate’s own checks offline: refusing an archive that matches neither hash, failing on an error while passing on warnings, and keeping the provider pins inside the module’s declared ranges.

Whether any of this actually blocks a merge

Section titled “Whether any of this actually blocks a merge”

Parity means both CIs run a gate. Whether a red gate can stop a merge is a repository setting, and today it does not: neither develop nor main has any branch protection — and main is the branch releases are cut from — so every gate in this table is advisory. A pull request can be merged with all checks red, and because the gate workflows are pull_request-only, a direct push to either branch runs none of them.

Terminal window
make check-branch-protection # reads the live setting via the GitHub API

The check derives the expected required-check list by parsing .github/workflows/*.yml for the job names GitHub turns into status-check contexts, rather than from a hardcoded list that would drift on the next rename. It then asserts protection is enabled, that every context a PR produces is required, that stale approvals are dismissed, that force-push and deletion are blocked, that an approving review is required, and that enforce_admins is on — without it an administrator can push straight past everything else. It also names the contexts that must stay advisory: the docs and dependency-manifest workflows are path-filtered, and Test Results is a check run an action creates behind an if:, so none of them reports on every PR and requiring one would leave a check pending forever and block every merge.

Three details are worth knowing about what it reads. Eleven of the thirteen shared gates live in developer-tests.yml and the other two — the SRT scan and the dependency audit — in security-checks.yml; because a context is a job, and those two workflows hold six and two jobs respectively, the thirteen gates produce eight requireable contexts. ⚠️ Requiring only the lint context is therefore weaker than it looks: the test suites are their own contexts, so all eight have to be required for a red test run to block a merge. It reads both enforcement mechanisms, classic branch protection and rulesets, because a branch can be fully governed by a ruleset while the classic endpoint reports nothing. And it distinguishes “not protected” from “cannot see”: the classic endpoint needs repository admin and answers 404 without it, so the tool cross-checks GET /repos/{slug}/branches/{branch}, which carries a protected boolean and is readable with plain pull access. --json therefore reports protected: null — not false — when the state genuinely could not be determined.

It is opt-in and blocks nothing: it needs network access and a token (pull access is enough to reach a verified answer about whether the branch is protected and to compare the required-check list, because GET /repos/{slug}/branches/{branch} carries a nested protection.required_status_checks object at that scope; administration:read is what the other five assertions — approvals, stale-review dismissal, force-push and deletion blocks, enforce_admins — need, and a run without it reports those five as unread, not as satisfied). One invocation reads one branch, so answering the question for this repository takes two — add BRANCH_PROTECTION_ARGS=--branch=main for the second. With no token or no network it exits 0 with an explanation; otherwise its steady-state result here is exit 1 with a single not_protected finding on each branch, which is the expected answer rather than a regression.

The absence of protection is a known, accepted residual, not an open task, and the decision is recorded in closed issue #933: enabling classic protection needs repository admin, which no contributor and no CI token here has. Nothing in the repository can substitute, because enforcement is server-side — a merge taken through GitHub’s own Merge button runs no code from this tree. So the condition for making this a required, blocking check, run with --fail-on-skip, is a repository setting changing: either somebody with repository admin enables protection, or an organization or enterprise owner publishes a branch ruleset targeting these branches, which needs no repository admin at all. Its own parsing and assertion logic is covered offline by scripts/tests/test_check_branch_protection.py.

Terminal window
make ui-test # Vitest, jsdom — no browser
make ui-build # lint + typecheck + production Vite build

Both CIs run the Vitest suite. Nothing in this layer opens a browser — layer 6’s UX review and first-time-user stories are the tiers that do.

If you are splitting a local run across lanes, note that these two share src/ui/node_modules with make lint-cicd. Both reinstall that directory with npm ci, which deletes it before rebuilding it, and so do the three UI targets lint-cicd performs itself (ui-lint, ui-build-only, codegen-check) — so “lint in one lane, UI tests in another” is two lanes writing one directory, which is how three gates once went red with three unrelated-looking missing-module errors. Every one of those targets now takes an exclusive lock on that directory, so the second lane waits rather than racing; it tells you it is waiting, and it gives up with an error after a bounded wait instead of hanging. The cost in a single serial run is one flock and no output. Neither CI is affected either way: both split these gates across separate runners with separate checkouts, so nothing there contends. See the comment above the UI section of the Makefile and scripts/ui_deps_lock.sh.

Terminal window
make srt-scan # Sample Security Review Tool: SAST over the checkout
make dep-audit # every pinned Python + Node dependency against OSV (fails on HIGH+)

Both run on every pull request in both CIs, and neither needs AWS. They cover different things and one does not imply the other: SRT’s syft stage builds an SBOM (inventory only, no vulnerability matching), so dependency CVEs are dep-audit’s job alone.

Findings are triaged in place rather than waved through: bandit findings take a line-scoped # nosec <ID> - <reason>, unreachable advisories go in scripts/security/dep_audit_allowlist.json with a justification (an entry that no longer matches any finding fails the audit, so delete it once the advisory is withdrawn or the package is upgraded), and security-matrix/Checkov findings go in scripts/srt/issues.json. Procedures: srt-security-scan.

dep-audit also prints an unmapped advisories section. GitHub publishes many advisories as unreviewed, with no affected package attached; matching by package and version, as OSV, npm audit and Dependabot do, can never report one. The section lists each unreviewed advisory from the last 60 days whose summary opens with a pinned package’s name and a range that includes the pinned version. It is a list to review, not a gate: it is read from prose, and it never changes the exit code. It needs a GitHub token (GITHUB_TOKEN, GH_TOKEN or a gh login), and says “not checked” without one; make dep-audit DEP_AUDIT_ARGS=--no-unmapped skips it. GitLab’s dep_audit job runs it. GitHub’s passes --no-unmapped, because the Actions token’s hourly request limit is shared by every job in the repository and one run of this report uses about a third of it.

An advisory in that list is not allowlisted: there is no OSV finding for the entry to suppress, so it would fail the audit as stale. Once reviewed it needs no record; it leaves the list when it is 60 days old or when GitHub maps it to a package, at which point OSV reports it normally.

Published, redacted snapshots of the four security tests per release live in security/test-results/<version>/, produced by make security-results.

The GitLab pipeline’s integration_tests job deploys a stack and drives fourteen numbered steps through it — default config, BDA mode, rule validation, concurrent batch processing, Test Studio evaluation, agentic extraction on a large table, single- and multi-document discovery, test comparison, API RBAC, IAM permissions boundary, and pipeline hooks. It needs AWS credentials, so it is GitLab-only: a change merged through a GitHub pull request has not run it.

⚠️ It runs nightly, not per-merge-request, so a green MR pipeline does not mean the deploy works. It was automatic on develop and on non-Draft MRs, which put 62–113 minutes onto every pipeline after a 45-minute check stage. It is now the nightly schedule, a weekly variant-probe schedule (below), and a manual play button, available on any MR targeting develop and on a develop pipeline.

The nightly runs the primary suite only. The deployment-variant probes (ZAP DAST, tenant isolation and the deploy variants in section 6) run on a separate weekly schedule, so a probe failure is never read as a nightly regression. The schedule itself is a GitLab project setting (Build › Pipeline schedules, carrying IDP_RUN_PROBES=true), not something in this repository; the probes run only while it exists and is active. Two more weekly schedules, while they are active, run the primary suite in eu-central-1 and ca-central-1; each result is labelled with its region, in the same Slack channel. A second nightly schedule, while it is active (carrying IDP_IAC=terraform), runs the primary suite against a Terraform deployment of the accelerator instead of a CloudFormation stack, labelled NIGHTLY TERRAFORM RUN. It is told apart from the CloudFormation nightly by that variable alone, so it leaves the UX review stack and the nightly first-time-user stories to the CloudFormation nightly, and the two runs queue for the same deploy lock rather than overlapping. To run that mode by hand, set IDP_IAC=terraform for the deploy driver, scripts/sdlc/codebuild_deployment.py; how it differs from the CloudFormation run, and the CI pipeline-stack update it needs before the schedule is turned on, are in the coverage document below.

Two things follow. Click the button before merging anything that touches the deploy path — template.yaml, publish.py, patterns/, nested/, src/, lib/, config_library/, feature-platform/, iam-roles/, scripts/ — because it is the only pre-merge signal this tier has left. And because a breaking change can merge and surface a night later (a week later, for a variant probe) with other merges on top of it, a scheduled failure is triaged by bisecting the merges since the last green run, not by reading the pipeline’s own commit. The trade is written out in full in scripts/sdlc/docs/CI_TEST_COVERAGE.md.

Per-step detail, what each step asserts, and how to reproduce a single step by hand are in scripts/sdlc/docs/CI_TEST_COVERAGE.md.

6. Live-stack tiers (manual, plus the weekly probe schedule)

Section titled “6. Live-stack tiers (manual, plus the weekly probe schedule)”

Each of these needs a deployed stack (or deploys its own) and has a written procedure, and each but the first-time-user stories is mandatory for a release. Several also run in CI in some form, on GitLab only, so a GitHub pull request covers none of them: the dynamic API RBAC matrix is step 12 of the GitLab integration_tests deployment (scripts/sdlc/codebuild_deployment.py shells out to make api-test against the stack that job deploys); the ZAP DAST, tenant isolation and deploy-variant rows run on the weekly variant-probe schedule, a scheduled pipeline of its own separate from the nightly; the UX review runs unattended as the advisory ai_ux_test job on merge requests that change src/ui/; and the first-time-user stories run as advisory jobs on merge requests that change CHANGELOG.md and every night against the review stack. Note when integration_tests runs: automatically only in the scheduled pipelines; on a merge request to develop and on develop itself it is a manual button, and on any other branch it does not run, so nothing exercises a change before it merges unless someone presses that button. The rest of the rows run only when a person asks.

TierWhat only a live stack can proveCommandProcedure
API RBAC — dynamic (also runs in the GitLab integration_tests job)that the deployed resolver enforces authorization per Cognito group and configuration-profile scope, not just that the code looks rightmake api-test STACK_NAME=…api-rbac-test
Cognito authorization behaviourthat the pre-token group-mapping trigger and client attribute permissions behave as the docs claim — they do not alwaysmake live-auth-checks · make verify-idp-federationlive-auth-checks
UX review (browser)that a person can actually complete each flow in the web UI, and how it feels doing so — functional pass/fail plus usability findings. One of the two tiers here that open a browser, beside the first-time-user stories belowmake ux-test STACK_NAME=…ux-test
First-time-user stories (browser)that a person new to the product can do what the published documentation says they can, by following it literally — each story is one intent attempted as one persona and judged against what that persona’s role should be able to do, with every hesitation, dead end and gap between the docs and the screens recorded as it goes. Advisory in GitLab CI on merge requests and nightly; not a release tiermake ai-ux-test-stories-local STORY=…ux-test-stories
Inbox end to endthat a message posted through idp_common.notifications (direct, Cognito group fan-out, idempotency key, dedupe key) is the one the deployed API lists, counts and marks read for its recipient, and that another user cannot reach it. Creates and removes a temporary Cognito group, two users and their Inbox itemsmake stacktest-inbox STACK_NAME=…run-stack-tests
Review queues end to endthat a document routed under a profile with review queues on reaches the deployed queue, can be claimed, edited and completed through the API with a real Annotator token, and finishes with the final status object, an unchanged machine manifest and a final manifest. Creates and removes a temporary profile, Annotator, Admin and document, and lists that Annotator on the Default queue for the duration of the run (CHECK_EVENTS=1 adds a temporary EventBridge rule and SQS queue)make stacktest-review-queues STACK_NAME=…run-stack-tests
ZAP DAST (also on the weekly probe schedule, while it is active)that the deployed HTTP surface has no exploitable findingmake stacktest-zap STACK_NAME=…run-stack-tests
Tenant isolation (also on the weekly probe schedule, while it is active)that a stack deployed with TenantIsolationMode=Strict keeps two tenants apart: each tenant’s users read their own document and are refused the other’s, strict-mode ingest refuses an upload outside every tenant’s prefixes, and the generated per-tenant access points refuse a cross-tenant upload. Creates and removes throwaway profiles, tenants, Cognito users, two IAM roles and an access point stackmake stacktest-tenancy (self-deploys a Strict stack) · make verify-tenant-isolation STACK_NAME=… (an existing Strict stack)run-stack-tests
Deploy variants (also on the weekly probe schedule, while it is active)that each hosting/parameter combination actually creates and servesmake stacktest-hosting-global · make stacktest-waf · make stacktest-hosting-private (VPC) · make stacktest-jobsapi (VPC) — list them with make stacktest-listrun-stack-tests
Template transformsthat a transformed template deploys and processes a real document — the only tier that canmake transform-deploy-test-headless · make transform-deploy-test-govcloud · both: make transform-deploy-test-alltransform-deploy-test
Seller Entitlement Service e2ethat the service deploys into a seller account, grants correctly, and refuses correctlymake stacktest-sellerrun-stack-tests
Analytics interpreter network reachwhat a session on the AWS-managed Bedrock AgentCore Code Interpreter the analytics agent runs Python on can reach: public DNS and HTTPS, a raw TCP connection, Amazon S3 in the session’s region and in another region, the regional STS endpoint, a PUT to a presigned URL (and whether the object arrives), and whether any AWS credential source is present. Compares each with the posture recorded in scripts/sdlc/sandbox_network_probe.py and prints a pass/fail table. Needs no stack; creates and deletes a throwaway bucketmake stacktest-sandbox-networkrun-stack-tests
Terraform property paritythat a Terraform deployment of the release stores the same configuration as a CloudFormation deployment of it with the same parameters, every resource of the main and nested stacks read back through Cloud Control, beyond the explained differences in terraform/tools/parity_baseline.json. Read-only; needs both deploymentsmake terraform-parity CFN_STACK=… TF_STACK=…release-validation (tier I)
In-place upgrade (X→Y)that an existing customer stack survives update-stack without rollback and still works afterwardsmake stacktest-upgrade (pointer)test-upgrade
Migration from genai-idp-terraform (also on the weekly Terraform-migration schedule, while it is active)that a deployment of the separate awslabs/genai-idp-terraform implementation, at a pinned tag and seeded with documents, users, a configuration edit, a baseline and a test set, moves onto this repository’s module with every phase of idp-cli terraform migrate-from-genai-idp-terraform, keeping every object and item, and that afterwards both users sign in to the new deployment with their old passwords and keep their scope, a new document is processed by the new deployment only, and destroying the old deployment leaves the adopted resources standing. Deploys and destroys both deployments: measured end to end at 166 minutes, 76 of them waiting on DynamoDB’s two time-to-live intervals and 33 the adoption apply and writes result.json and summary.txt in its work directorymake stacktest-terraform-migration OLD_NAME=… NEW_NAME=… TERRAFORM_MODULE_URL=…release-validation (tier J)

make ux-test STACK_NAME=… prepares a throwaway Cognito user and a session, then a person or the assistant drives the flows in scripts/ux_flows.yaml in a real browser and reports functional pass/fail per flow plus ranked usability findings. It is deliberately not a script: a shell script can assert that a page returned 200, not that the page made sense.

The review can also be recorded as a narrated, captioned video, so a finding can be shown to the team instead of re-demonstrated live:

Terminal window
make ux-record-deps # ffmpeg, ffprobe, boto3, Pillow
./scripts/ux_recorder.py start --stack <STACK> --persona Admin --url-contains cloudfront \
--say "We start on the Test Studio sets tab."
./scripts/ux_recorder.py mark "Open the annotation queue" --say "We open the queue …"
./scripts/ux_recorder.py stop --say "That ends the review."
AWS_PROFILE=default ./scripts/ux_recorder.py render --voice Ruth # --dry-run first

The recorder attaches a second DevTools session to the tab being driven, captures frames only when the screen changes, and re-paces the result for a viewer: idle gaps clamped, paused stretches dropped, narration spoken by Amazon Polly, every click drawn on the frame that was on screen when it happened. Output — review.mp4, review.srt, segments.json, review.md — lands under gitignored scratch/ux-recordings/, in a session directory that also holds the tab’s address and title and a click log (events.jsonl, timeline.json) that labels clicks with text from the page. A recording of a live stack shows real documents and nothing is redacted: never commit the session directory or attach any of it to a pull request. One exception is a finished product-demo video its author chooses to publish: the author uploads that demo.mp4, and nothing else from the session, to get its asset URL for the demo videos page. The MR UX test’s mp4 of the CI review stack, which holds only shipped example data, is the other. What a click label can hold is set out under Privacy in ux-test. Details: scripts/README.md.

On a merge request the same review runs unattended. When a non-Draft MR to develop changes src/ui/, the advisory ai_ux_test job tests the MR’s UI build in a confined headless browser against the long-lived review stack idp-ux-nightly, and appends its findings and a narrated video to the automated code-review comment for that commit. It gates nothing. make ai-ux-test-local MR=<iid> runs the same test in Docker as a dry run, and make ai-ux-browser starts only the confined browser, with DevTools on 127.0.0.1:9222, for an interactive session on a machine with no local Chrome. Contract: ux-test-ci.

scripts/ux_stories.toml holds stories that say only what a person wants, each with the personas it runs as and whether each persona’s role should be able to do it, be refused, or not be offered it at all. A model that knows the product only through this documentation attempts each story in a confined headless browser, searching and reading these pages through tools of its own and noting every hesitation and dead end as it goes. Every tool call is journalled, and the verdict — done, stuck, refused as the docs say, offered a control the docs say the role lacks, and so on — is computed from that journal and the persona’s expected access, never from the model’s prose, so one run of a story can be compared with the next. Each story is told to change nothing on the shared review stack, and a best-effort guard in front of the browser refuses the saves it recognises: a story whose task ends in a save stops at that step and says so.

make ai-ux-test-stories-local STORY=<id> runs a story in Docker as a dry run (MR=<iid> runs the stories that MR’s added CHANGELOG entries select, and INTERACTIVE=1 lets you watch it live), make ai-ux-test-stories-pair starts the confined browser and prints the command for a Claude session you start to play the user, and make ai-ux-test-stories-list and make ai-ux-test-stories-search Q="…" are the authoring loop. In CI, the advisory ai_ux_test_stories job runs the stories a non-Draft MR’s added CHANGELOG entries select and replies in the code-review thread, and every night the nightly set runs and feeds a digest of regressions; none of it gates anything. Procedure: ux-test-stories; unattended contract: ux-test-stories-ci.

make benchmark-release VERSION=… PREV=… runs the release-vs-release A/B — accuracy, completeness, cost, latency and confidence calibration against the previous published release. The wider suite in benchmarks/ (a config × document-size matrix with exact ground truth) is what produces the published Configuration Guidance and is the gate for any change that could move accuracy or cost.

Start at the Benchmarking Guide; read benchmarks/matrices/METHODOLOGY.md before writing down a number, and the run-benchmarks procedure before running one.

A test that leaves no record cannot be cited later, so three of the layers above write one:

RecordWhat it holds
Release Validation Recordsone file per release, never overwritten: every tier that needs a live stack, its verdict, and what it found
security/test-results/<version>/redacted per-test snapshots of SRT, ZAP DAST and RBAC static + dynamic
Release Benchmark Audit Trailthe release-vs-release A/B, with n and spread on every claim

Everything else — raw logs, probe output, UX recordings — stays in gitignored scratch/. Records are public-safe by construction: account ids, VPC/subnet/security-group ids, API hostnames, pool ids, stack physical ids and local paths are replaced with placeholders.

One release validation exercises every layer on this page in a fixed order, on real stacks, and writes the three records above. It is a two-day job, driven by the release-validation procedure — the umbrella over every per-tier procedure linked here. The per-tier make targets stay runnable on their own, which is how you validate a single change without validating a release.