Testing
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
Testing
Section titled “Testing”Every test layer and tier in this repository: what it proves, how to run it, whether
CI runs it for you, and where its results are recorded. This is a map of the tiers,
not an index of individual test functions — there are thousands of those across
hundreds of test modules, and one added inside a suite that already runs needs no
change here. make test-list enumerates the suites themselves, and the section on
what make test does not run is the one place the
suite-level exceptions are written down. If you are looking for what a given
release was actually validated against, that is the
Release Validation Records; this page describes the
methods, not any particular run of them.
The organising fact: most of what protects this repo runs on every pull request, but the tiers that need a deployed stack cannot. Roughly a dozen methods below run only when a person asks for them, which is why each one has a written procedure and a record of its last run.
The layers at a glance
Section titled “The layers at a glance”| Layer | Needs AWS? | Runs in CI? | Entry point |
|---|---|---|---|
| 1. Offline test suites | no | ✅ both CIs, but through two other targets — see below | make test |
| 2. Static gates | no | ✅ both CIs | make lint-cicd · make typecheck · make terraform-check |
| 3. Web UI unit tests | no | ✅ both CIs | make ui-test |
| 4. Security scanning | no | ✅ both CIs | make srt-scan · make dep-audit |
| 5. Integration smoke suite | yes | ⚠️ GitLab scheduled (primary suite nightly) + manual button — not a merge gate | pipeline integration_tests |
| 6. Live-stack tiers | yes | ❌ manual, except the ZAP DAST and deploy-variant rows (GitLab weekly schedule, while it is active) | see the table |
| 7. Benchmarks | yes | ❌ manual | make benchmark-release |
make test and make lint are the two commands to run before opening a pull
request. make all is both.
1. Offline test suites
Section titled “1. Offline test suites”make test discovers and runs every non-integration suite in the repo — the
idp_common library, idp_cli, idp_sdk, idp_feature_sdk, the feature platform,
the per-Lambda suites, the config library, and the repo’s own tooling tests under
scripts/, with the handful of documented exceptions below.
make test-list prints the discovered roots without running them, which is the honest
answer to “is my new suite actually being run?”.
make test # everything, auto-discoveredmake test-list # which roots were discoveredcd lib/idp_common_pkg && make test-unit # one root, isolatedmake test-integration-all # only the integration-marked tests (needs AWS)make test itself runs in neither CI. CI runs the same suites through three other
targets — make test-cicd -C lib/idp_common_pkg, make coverage-all-cicd, which runs
every suite a coverage tree measures, and make test-packages-cicd for the rest — and
the last of those is a hand-enumerated recipe, so a suite that make test
discovers but nobody adds to it runs locally and on no pull request. 22 registered
roots holding 506 tests were in exactly that state, including the resolver suites
covering the download allow-list and the discovery upload target.
scripts/tests/test_src_lambda_tests_in_ci.py now closes that by deriving both
sides — every directory holding a tracked test_*.py, against the paths the two
recipes run pytest against plus the directories the coverage trees measure — so a root that is in neither is a test failure
rather than a gap somebody finds by hand months later. A suite that genuinely cannot
run in the shared gate goes in the excluded registry below
with a reason.
Both of those targets run every suite with the AWS environment removed — no
region, no credentials, no profile, the shared AWS config file neutralised and the
instance metadata service disabled — because that is what a CI runner supplies. The
wrapper is defined once, in make/hermetic_aws.mk, and included by the root
Makefile and by lib/idp_common_pkg/Makefile; make test-integration there
deliberately does not use it, because integration tests need real credentials.
A suite that needs a region or placeholder credentials must therefore supply them
in its own conftest.py, where os.environ.setdefault reinstates them after the
wrapper has taken the machine’s real ones away. Inheriting either from the shell
means the suite passes where it was written and fails where it is gated, and there
are two distinct ways that happens:
- A region. A Lambda handler the suite imports builds a boto3 client at module
scope, so importing it needs a resolvable region and botocore raises
NoRegionErrorduring collection without one. A developer machine supplies one from the shared AWS config file; a runner supplies nothing. - Credentials. botocore freezes the session’s credentials object into a client
when the client is constructed, so a client built at import time with none
resolvable can never sign — no matter how many credentials appear later, including
the ones
motosets when a fixture starts. A developer box on EC2 resolves credentials from the instance metadata service without anyone noticing; a runner cannot reach it. The symptom isAttributeError: 'NoneType' object has no attribute 'access_key'out ofbotocore/auth.py, which does not mention credentials at all.
A corollary worth knowing: anything a test module does to os.environ at import
time affects the whole session. -m "not integration" deselects tests after
collection has imported their modules, so even a file whose tests never run can
change the environment every later module is imported under.
scripts/tests/test_offline_suites_are_hermetic.py checks all of this, and checks
the stripping wrapper itself by measurement: it builds an environment that supplies
a region and credentials through every source, puts it through the wrapper, and
requires that neither survives.
There is no standing failure set — Expected standing failures: 0 on a
correctly installed tree, and the enumerated list of accepted failures in
full-test-battery
is empty. So treat any failure as a real regression until proven otherwise.
scripts/tests/test_standing_failure_baseline.py holds that claim, this page and the
two skills that repeat it to the same number, so they cannot drift apart again, and
make test itself compares the failures it observed against that table — a failure
it does not declare and a declared row whose test passed both fail the run.
Most surprising failures are still a stale virtualenv missing the pinned [test]
extras — but do not expect a broken install to announce itself as an
ImportError. Several Lambdas catch a missing idp_common on purpose and degrade
(feature-platform/main-stack-extensions/lambdas/apply_feature_config_preset/index.py
logs at ERROR and applies a config preset without recording a revision), so a
suite that exercises the non-degraded path fails on a bare assertion instead. Two
tests in test_apply_feature_config_preset.py
(test_remove_hands_the_pipeline_back_to_default_then_deletes and
test_remove_keeps_an_active_profile_when_there_is_no_default_to_fall_back_to) were
misread as a standing failure of this repo for exactly that reason. With
idp_common unimportable that file reports 2 failed, 18 passed; with the pin the
make targets export (PYTHONPATH naming every lib/* package root, absolute) it
reports 20 passed, on the same interpreter and the same commit. Measured identically under Python 3.12 and
3.13, so it is not a version incompatibility. Check that
python3 -c "import idp_common; print(idp_common.__file__)" resolves inside your own
checkout before reading anything else — an editable install can silently point at a
different, or deleted, checkout. The diagnosis
order, the per-suite expected totals, and how to prove a failure is inherited are in
the full-test-battery
procedure. Conventions for writing tests — pytest markers, moto, conftest
layout — are in
testing-qa.
Suites make test does not run
Section titled “Suites make test does not run”Auto-discovery means a suite cannot be forgotten, not that every suite is run.
scripts/run_all_tests.py refuses to run at all when it finds a directory holding a
test_*.py that is in neither of its two registries — the roots it runs, and the
roots it excludes with a written reason — so tests in a new location cannot be
silently skipped. An exclusion covers only the directory named: nesting under an
excluded directory used to inherit the exclusion, which meant excluding scripts
quietly accepted every future test directory beneath it, so each excluded directory
is now listed on its own. That check backs make test, which runs in neither CI, so
scripts/tests/test_testing_doc.py re-derives it on every pull request, where
pytest scripts/tests does run.
These are the directories in the excluded registry. They are listed because a suite that exists and never runs is otherwise indistinguishable from one that passes:
Not run by make test | Why |
|---|---|
scripts | scripts/test_api_rbac.py is the live RBAC harness driven by make api-test against a deployed stack (layer 6), not a pytest suite; collecting it picks up its test_email() helper as a test |
src/lambda/ocr_benchmark_deployer | test_local.py needs huggingface_hub, which is not a test dependency |
samples/lambda-hook-inference/GENAIIDP-chandra-ocr-hook | test_local.py is a manual local-run script and collects zero pytest tests (measured) |
lib/idp_sdk/idp_sdk/_core | source, not tests: test_studio_processor.py is the Test Studio processor module, which the test_ prefix makes look like a suite |
lib/idp_common_pkg/manual_tests/agents | operator-run scripts, not a suite: each one drives real Bedrock, Athena or DynamoDB against a deployed stack and bills model calls. Run by hand (python manual_tests/agents/test_analytics.py -q "…"); norecursedirs in lib/idp_common_pkg/pytest.ini keeps a bare pytest from collecting them |
Adding an exclusion, or lifting one of these, fails that guard until this table and the registry agree — it is checked in both directions, so a row that outlives the exclusion it describes fails too.
The integration-marked tier runs in no CI
Section titled “The integration-marked tier runs in no CI”Separate from the directory exclusions above, the tests marked
@pytest.mark.integration are deselected from every suite on this page. The
mechanism is the runner, not a config file. scripts/run_all_tests.py runs one
pytest -m "not integration" <root> subprocess per root, so the filter applies to
every root it discovers — including a new package under lib/, which inherits
nothing. Only lib/idp_common_pkg/pytest.ini carries addopts = -m "not integration" of its own; lib/idp_sdk and lib/idp_cli_pkg have an empty
addopts, and the root pytest.ini deliberately sets none, because an addopts
there would silently change collection for every suite that resolves to it.
Those tests call live AWS — real Bedrock, Textract, S3 and DynamoDB rather than
moto — and within idp_common they are the only ones that do. (Layers 5 to 7
below need AWS too, but none of them is a marked pytest suite.) They run with
make test-integration-all, or per-package with make test-integration, and
nothing else runs them: not make test, not GitHub, not GitLab. A person
typing the target is the whole of their coverage.
⚠️ GitLab’s integration_tests job is a different thing with a similar name.
It is the deploy-driven smoke suite in layer 5
— a CodePipeline stack deployment — and it never invokes pytest -m integration.
Reading the stage name as coverage of this tier is what let 14 of its 23 tests sit
failing on tooling faults until a release validation found them
(#1307):
the tier was uncovered, and the exemption recording that it was local-only cited
the stage as the reason it did not need to be. test_ci_gate_parity.py’s
test_the_integration_pytest_tier_runs_in_no_ci now measures the absence instead
of asserting the stage name, so if a CI starts running the tier, the stale
exemption fails rather than the claim going quietly out of date.
Two consequences worth keeping in mind when adding one of these tests. The marker,
not the directory, is what makes a test part of this tier — make test-integration
runs pytest -m "integration" over its whole package tree, so an
integration-marked test
under tests/unit/ is in it. And because nothing gates them, a broken one stays
broken: a test whose seams are all mocked needs no credentials and belongs in the
default gate, where something will notice.
Making sure a run tests this checkout
Section titled “Making sure a run tests this checkout”idp_common and the SDKs are editable installs, so import idp_common reads
whatever pointer is in the active interpreter’s site-packages — not necessarily this
checkout. Where more than one checkout is worked on at once and python3 resolves to a
shared interpreter rather than a per-project virtualenv, that directory is shared and the
last pip install -e wins for all of them. make test-cicd is itself a writer: its
test-unit-cicd target runs pip install -e ".[test]" unless SKIP_INSTALL=1 is set,
so running the gate repoints the pointer as a side effect.
Nothing raises when this happens. The imported package is real and self-consistent, just a different revision, so it shows up as an unrelated-looking assertion failure or as a green run whose coverage number describes another tree.
Through make, the pin is applied for you. PYTEST_HERMETIC in
make/hermetic_aws.mk — the wrapper every pytest invocation in both Makefiles goes
through — exports an absolute PYTHONPATH naming every first-party root of the checkout
the makefile belongs to, and scripts/run_all_tests.py passes the same value to each of
the pytest subprocesses make test starts. The set is derived from
lib/*/pyproject.toml, the same rule that decides what FIRST_PARTY_EDITABLES installs,
so a package added under lib/ is pinned without anyone adding it anywhere;
scripts/tests/test_first_party_pythonpath.py asserts the three implementations of that
rule agree, and measures the wrapper by importing all five packages under it. A caller’s
own PYTHONPATH is kept, after the pin. make test FIRST_PARTY_PYTHONPATH= suppresses
it, for deliberately testing an installed copy.
Running pytest directly, pin it yourself — every root, and absolute. The packages
import each other, so a pin naming lib/idp_common_pkg alone is refused by the next one;
and a relative pin is dropped by any subprocess started in another directory, which
several suites do start. scripts/tests/first_party_provenance.py is what refuses, and it
is called from lib/idp_common_pkg/tests/conftest.py,
feature-platform/main-stack-extensions/tests/conftest.py, scripts/tests/conftest.py,
scripts/tests/test_model_surface_consistency.py and
lib/idp_sdk/tests/unit/test_config_operations_region.py. It compares checkout
identity, deriving each side’s root by walking up to .git, so a git worktree validates
against itself and passes while a worktree nested inside another checkout is correctly
refused. Before refusing it tries to repair: it puts this checkout’s own roots first on
sys.path and reports having done so, since that fixes the process in hand and not the
environment. It still refuses when the package was already imported — re-importing one
other modules hold references to leaves two live copies — and when this checkout has no
copy to prefer. A refusal is not a failing test; it is a run that did not happen, and
the message says so, because a collection error at the end of a long log reads as “some
tests failed”.
Fix it durably by installing with the interpreter you actually want
(<your-venv>/bin/python -m pip install -e "lib/idp_common_pkg[test]", by path and never
by bare name — see dependency-confusion.md).
scripts/check_first_party_deps.py, which runs in both CIs and after every make setup,
reports an editable pointer into another checkout as a failure — it answers “from source,
and from which source”, where answering only the first half is how this condition stayed
invisible while the control that looked like it covered it was green.
IDP_ALLOW_FOREIGN_FIRST_PARTY=1 downgrades both that failure and the pytest-side
refusal to a note when you are deliberately testing an installed copy; it is a
per-invocation switch, so it is not in scripts/tests/gate_exemptions.json, for the
reason that file records for ALLOW_SHARED_BRANCH.
2. Static gates (lint, types, and hand-written scanners)
Section titled “2. Static gates (lint, types, and hand-written scanners)”make lint is the local gate; make lint-cicd is the same set in check-only mode
and is what both CIs run. Several members of the set are not linters at all but
hand-written scanners for classes of defect that have each shipped at least once:
| Gate | What it catches |
|---|---|
make ruff-lint · make format | Python style and formatting (88 cols, Python 3.12) over every tracked .py file except a named per-file debt list |
make check-lint-debt | that debt list staying honest: an excluded file that gained a finding, one that is now clean and should be delisted, a dead path, a bare directory name in one of the three generated arrays, or a tracked file ruff’s walk never reaches (which is what covers the top-level exclude array, whose bare names are deliberate). ruff.toml excluded five bare directory names until #975, which match at any depth, so 442 of 1230 tracked .py files were read by neither gate. Use python3 scripts/check_lint_debt.py --explain <path> to ask whether one file is linted — no ruff invocation answers that correctly |
make check-ui-i18n-catalog | the UI’s committed English message catalog (src/ui/src/i18n/en.json) drifting from the defineMessages calls it was extracted from, and a translation catalog that would fail the UI build (malformed ICU, an argument or plural the English does not have, an id the English no longer has). Check-only: it re-extracts into a temporary file and diffs; make ui-i18n-extract is the writer. Without src/ui/node_modules, or on a machine the installed @formatjs/cli ships no native binary for, it reports a skip, never a pass; in CI either is a failure (#1770) |
make check-ui-string-debt | a hardcoded user-facing string added to the UI: a file carrying more findings than scripts/ui_string_debt.json pins for it, a file not in that baseline carrying any (a new file and a converted file both start at zero), an entry that is now vacuous or names a vanished path, or a counting configuration (src/ui/eslint.i18n.config.js) whose hash is not the one the counts were measured with, so a widened exclude list cannot lower every count silently. Strings inside const ALL_CAPS = ... declarations are not counted (a declared gap, with its numbers, in the ratchet’s docstring). Use python3 scripts/check_ui_string_debt.py --explain <path> to see one file’s findings; --write re-records a smaller count and refuses a larger one without --allow-new-debt "<reason>" (#1770) |
make cfn-lint | any file declaring AWSTemplateFormatVersion — discovered by content, so a new template cannot escape it. Fails on errors only |
make check-arn-partitions | hardcoded arn:aws: / amazonaws.com instead of ${AWS::Partition} / ${AWS::URLSuffix} — GovCloud compatibility |
make check-filtered-scans | DynamoDB Scan with a filter expression that cannot see all matches |
make check-data-plane-tags | the idp:plane=data tag on the Lambdas that must carry it |
make check-markdown-links | a relative Markdown link that resolves to nothing, an #anchor naming a heading that has since been retitled, a published docs/ page linking relatively to one docs-site/setup.sh does not symlink (resolves on GitHub, 404s on the site), and a code fence that swallows content, which would otherwise leave this gate blind to the rest of the file. Every tracked .md file, from git ls-files. Anchors are slugified by github-slugger’s rule, which scripts/tests/test_markdown_links.py verifies against the real package over every heading in the tree. External http(s) URLs are not fetched, so a green result says nothing about them (#1068) |
make validate-buildspec | malformed CodeBuild buildspecs, and a loop in a multi-line command that cannot report its own failure — otherwise a deploy-time failure |
make codegen-check | generated GraphQL types drifting from the schema |
make typecheck | basedpyright over every tracked .py file — this is what both CIs run. make typecheck-pr narrows it to the files a branch changes, for local latency, and is a gate in neither |
make api-test-static | an API operation added without authorization, and drift between the dispatcher’s generated required-groups manifest and scripts/api_rbac_expectations.yaml — see layer 6 for the live half |
python3 scripts/check_first_party_deps.py | a first-party package in the current environment that came from a package index rather than from lib/, which on public PyPI is somebody else’s code |
scripts/tests/test_doc_install_commands.py (part of make coverage-all-cicd) | a pip install documented in a fenced code block that could resolve a first-party name from an index — a bare name, or a path install missing a sibling the package requires by name. The environment checker above cannot see an instruction nobody has run yet |
python3 scripts/sdlc/validate_service_role_permissions.py | the CloudFormation service role missing a permission the templates need |
scripts/tests/test_agent_capabilities.py (part of make test-packages-cicd) | an agent, tool, MCP tool or claude -p harness with no entry in scripts/tests/agent_capabilities.yaml, an entry whose surface is gone, and an entry that takes document-derived or external text and reads private data with an external output that no named control limits and no recorded finding lists. Surfaces are derived from source by AST. It checks the manifest’s declarations, not whether they are true; the agent-capability-review skill is the audit that re-reads them. scripts/tests/test_analytics_role_grants.py bounds the IAM grants of the roles those entries name |
scripts/tests/test_threat_model_evidence.py (part of make test-packages-cicd) | an entry under security/threat-modeling/ (every one the export builder reads) whose mitigations neither cite a test, gate or make target that resolves nor say UNVERIFIED, and a rise in the pinned count of UNVERIFIED entries |
Three of these gates guard the gates themselves:
scripts/tests/test_ci_gate_parity.py fails if a gate runs in one CI and not the
other, or if lint-cicd becomes weaker than local make lint;
scripts/tests/test_nested_stack_parameters.py checks parent-to-nested stack
parameter wiring that cfn-lint’s own rule cannot see; and
scripts/tests/test_lint_debt_gate.py drives check-lint-debt through each
failure mode it claims, because a ratchet nobody has watched fail is not a
ratchet. All three exist because every gap they cover was originally found by
hand, months late.
make typecheck reads every tracked .py file — git ls-files '*.py' | wc -l and
basedpyright’s filesAnalyzed agree exactly, and that identity, rather than any
particular count, is what scripts/tests/test_pyright_config.py asserts by deriving
the set from git ls-files rather than from a list. Run those two commands for
today’s figure rather than looking for one on this page: it changes with almost every
merge, and a written-down count has gone stale in three separate documents at once.
The include array once named six paths and reached 432 of the 1,230 tracked at the
time, and two NameError-class defects reached develop through the gap.
Reading every file is a weaker property than it sounds, and the same suite now
covers the difference. basedpyright honours PYTHONPATH, which neither the
make target nor either CI sets, so with reportMissingImports configured "none"
the shared idp_common library did not resolve and no call into it could produce a
diagnostic — zero errors over a file count that matched git ls-files exactly,
hiding eleven real ones (#1109).
pyrightconfig.json’s extraPaths fixes resolution in the configuration rather
than the environment, using relative paths so it cannot resolve against another
checkout, and the suite asserts that first-party imports really do resolve — by
running basedpyright, not by inspecting the JSON.
The Terraform offline gate
Section titled “The Terraform offline gate”make terraform-check is a static gate that is not part of make lint, because it
needs network access and about two gigabytes of cache. Each CI runs it as its own job,
terraform_checks. It works in four steps:
- It translates
template.yaml, with the nested templates read from source, into a temporary directory usingterraform/translatorin--offlinemode. No AWS call is made. - It installs each binary in
TERRAFORM_CHECK_BINARIESinto a cache. By default these are the floor and latest-stable releases of both Terraform and OpenTofu, pinned in theMakefileand mirrored in both CI configs. An archive is accepted only when its SHA256 matches both the release’s publishedSHA256SUMSand the hash committed inscripts/terraform_binaries.lock.json. - It runs
init -backend=falseandvalidateon the generated module with each binary. Provider versions are pinned exactly through an override file:awsandtimefrom the lock file, andawsccto the vendored schema’s version. Errors fail the gate. Warnings are counted and printed but do not fail it. - It runs
make terraform-scan-diff: a pinnedcheckov(CHECKOV_VERSIONin theMakefile) over the CloudFormation templates and over the generated module, failing on a Terraform finding the CloudFormation side does not have unlessscripts/terraform_scan_baseline.jsonexplains it. Findings on checks with no CloudFormation counterpart are pinned per check and resource type inscripts/terraform_scan_tf_only_baseline.json, so a new one fails too.
The gate proves that the generated HCL parses, that its references resolve, and that
every resource and attribute type-checks against the pinned provider schemas, on the
oldest and newest releases the module claims to support. It does not plan or
deploy anything, so a value AWS would reject at apply time, an IAM policy grammar
error, or an ordering problem between resources is outside what it can see. The
translator’s and bridge’s own unit tests run in make test (layer 1) and are not
repeated here. scripts/tests/test_terraform_check.py covers the gate’s own checks
offline: refusing an archive that matches neither hash, failing on an error while
passing on warnings, and keeping the provider pins inside the module’s declared
ranges.
Whether any of this actually blocks a merge
Section titled “Whether any of this actually blocks a merge”Parity means both CIs run a gate. Whether a red gate can stop a merge is a
repository setting, and today it does not: neither develop nor main has any
branch protection — and main is the branch releases are cut from — so every
gate in this table is advisory. A pull request can be merged with all checks red,
and because the gate workflows are pull_request-only, a direct push to either
branch runs none of them.
make check-branch-protection # reads the live setting via the GitHub APIThe check derives the expected required-check list by parsing
.github/workflows/*.yml for the job names GitHub turns into status-check contexts,
rather than from a hardcoded list that would drift on the next rename. It then
asserts protection is enabled, that every context a PR produces is required, that
stale approvals are dismissed, that force-push and deletion are blocked, that an
approving review is required, and that enforce_admins is on — without it an
administrator can push straight past everything else. It also names the contexts
that must stay advisory: the docs and dependency-manifest workflows are
path-filtered, and Test Results is a check run an action creates behind an if:,
so none of them reports on every PR and requiring one would leave a check pending
forever and block every merge.
Three details are worth knowing about what it reads. Eleven of the thirteen shared gates
live in developer-tests.yml and the other two — the SRT scan and the dependency
audit — in security-checks.yml; because a context is a job, and those two
workflows hold six and two jobs respectively, the thirteen gates produce eight
requireable contexts. ⚠️ Requiring only the lint context is therefore weaker than
it looks: the test suites are their own contexts, so all eight have to be required
for a red test run to block a merge. It reads both enforcement mechanisms,
classic branch protection and
rulesets, because a branch can be fully governed by a ruleset while the classic
endpoint reports nothing. And it distinguishes “not protected” from “cannot see”:
the classic endpoint needs repository admin and answers 404 without it, so the
tool cross-checks GET /repos/{slug}/branches/{branch}, which carries a
protected boolean and is readable with plain pull access. --json therefore
reports protected: null — not false — when the state genuinely could not be
determined.
It is opt-in and blocks nothing: it needs network access and a token (pull access
is enough to reach a verified answer about whether the branch is protected and to
compare the required-check list, because GET /repos/{slug}/branches/{branch}
carries a nested protection.required_status_checks object at that scope;
administration:read is what the other five assertions — approvals, stale-review
dismissal, force-push and deletion blocks, enforce_admins — need, and a run
without it reports those five as unread, not as satisfied). One invocation reads
one branch, so answering the question for this repository takes two — add
BRANCH_PROTECTION_ARGS=--branch=main for the second. With no token or no network it
exits 0 with an explanation; otherwise its steady-state result here is exit 1 with a
single not_protected finding on each branch, which is the expected answer rather
than a regression.
The absence of protection is a known, accepted residual, not an open task, and
the decision is recorded in closed
issue #933:
enabling classic protection needs repository admin, which no contributor and no
CI token here has. Nothing in the repository can substitute, because enforcement is
server-side — a merge taken through GitHub’s own Merge button runs no code from this
tree. So the condition for making this a required, blocking check, run with
--fail-on-skip, is a repository setting changing: either somebody with
repository admin enables protection, or an organization or enterprise owner
publishes a branch ruleset targeting these branches, which needs no repository
admin at all. Its own parsing and assertion logic is covered offline by
scripts/tests/test_check_branch_protection.py.
3. Web UI unit tests
Section titled “3. Web UI unit tests”make ui-test # Vitest, jsdom — no browsermake ui-build # lint + typecheck + production Vite buildBoth CIs run the Vitest suite. Nothing in this layer opens a browser — layer 6’s UX review and first-time-user stories are the tiers that do.
If you are splitting a local run across lanes, note that these two share
src/ui/node_modules with make lint-cicd. Both reinstall that directory with
npm ci, which deletes it before rebuilding it, and so do the three UI targets
lint-cicd performs itself (ui-lint, ui-build-only, codegen-check) — so
“lint in one lane, UI tests in another” is two lanes writing one directory, which
is how three gates once went red with three unrelated-looking missing-module
errors. Every one of those targets now takes an exclusive lock on that directory,
so the second lane waits rather than racing; it tells you it is waiting, and it
gives up with an error after a bounded wait instead of hanging. The cost in a
single serial run is one flock and no output. Neither CI is affected either way:
both split these gates across separate runners with separate checkouts, so nothing
there contends. See the comment above the UI section of the Makefile and
scripts/ui_deps_lock.sh.
4. Security scanning (SAST and SCA)
Section titled “4. Security scanning (SAST and SCA)”make srt-scan # Sample Security Review Tool: SAST over the checkoutmake dep-audit # every pinned Python + Node dependency against OSV (fails on HIGH+)Both run on every pull request in both CIs, and neither needs AWS. They cover
different things and one does not imply the other: SRT’s syft stage builds an SBOM
(inventory only, no vulnerability matching), so dependency CVEs are dep-audit’s
job alone.
Findings are triaged in place rather than waved through: bandit findings take a
line-scoped # nosec <ID> - <reason>, unreachable advisories go in
scripts/security/dep_audit_allowlist.json with a justification (an entry that
no longer matches any finding fails the audit, so delete it once the advisory is
withdrawn or the package is upgraded), and security-matrix/Checkov findings go in
scripts/srt/issues.json. Procedures:
srt-security-scan.
dep-audit also prints an unmapped advisories section. GitHub publishes many
advisories as unreviewed, with no affected package attached; matching by package
and version, as OSV, npm audit and Dependabot do, can never report one. The
section lists each unreviewed advisory from the last 60 days whose summary opens
with a pinned package’s name and a range that includes the pinned version. It is a
list to review, not a gate: it is read from prose, and it never changes the exit
code. It needs a GitHub token (GITHUB_TOKEN, GH_TOKEN or a gh login), and
says “not checked” without one; make dep-audit DEP_AUDIT_ARGS=--no-unmapped
skips it. GitLab’s dep_audit job runs it. GitHub’s passes --no-unmapped,
because the Actions token’s hourly request limit is shared by every job in the
repository and one run of this report uses about a third of it.
An advisory in that list is not allowlisted: there is no OSV finding for the entry to suppress, so it would fail the audit as stale. Once reviewed it needs no record; it leaves the list when it is 60 days old or when GitHub maps it to a package, at which point OSV reports it normally.
Published, redacted snapshots of the four security tests per release live in
security/test-results/<version>/,
produced by make security-results.
5. Integration smoke suite (CI only)
Section titled “5. Integration smoke suite (CI only)”The GitLab pipeline’s integration_tests job deploys a stack and drives fourteen
numbered steps through it — default config, BDA mode, rule validation, concurrent
batch processing, Test Studio evaluation, agentic extraction on a large table,
single- and multi-document discovery, test comparison, API RBAC, IAM permissions
boundary, and pipeline hooks. It needs AWS credentials, so it is GitLab-only: a
change merged through a GitHub pull request has not run it.
⚠️ It runs nightly, not per-merge-request, so a green MR pipeline does not mean
the deploy works. It was automatic on develop and on non-Draft MRs, which put
62–113 minutes onto every pipeline after a 45-minute check stage. It is now the
nightly schedule, a weekly variant-probe schedule (below), and a manual play button, available on any MR targeting
develop and on a develop pipeline.
The nightly runs the primary suite only. The deployment-variant probes (ZAP DAST,
tenant isolation and the deploy variants in section 6) run on a separate
weekly schedule, so a probe failure is never read as a nightly regression.
The schedule itself is a GitLab project setting (Build › Pipeline schedules,
carrying IDP_RUN_PROBES=true), not something in this repository; the probes run
only while it exists and is active.
Two more weekly schedules, while they are active, run the primary suite in
eu-central-1 and ca-central-1; each result is labelled with its region, in the
same Slack channel. A second nightly schedule, while it is active (carrying
IDP_IAC=terraform), runs the primary suite against a
Terraform deployment of the accelerator instead of a
CloudFormation stack, labelled NIGHTLY TERRAFORM RUN. It is told apart from the
CloudFormation nightly by that variable alone, so it leaves the UX review stack and
the nightly first-time-user stories to the CloudFormation nightly, and the two runs
queue for the same deploy lock rather than overlapping. To run that mode by hand, set
IDP_IAC=terraform for the deploy driver, scripts/sdlc/codebuild_deployment.py;
how it differs from the CloudFormation run, and the CI pipeline-stack update it
needs before the schedule is turned on, are in the coverage document below.
Two things follow. Click the button before merging anything that touches the
deploy path — template.yaml, publish.py, patterns/, nested/, src/,
lib/, config_library/, feature-platform/, iam-roles/, scripts/ — because
it is the only pre-merge signal this tier has left. And because a breaking change
can merge and surface a night later (a week later, for a variant probe) with other
merges on top of it, a scheduled failure is triaged by bisecting the merges since
the last green run, not by reading the pipeline’s own commit. The trade is written out in full in
scripts/sdlc/docs/CI_TEST_COVERAGE.md.
Per-step detail, what each step asserts, and how to reproduce a single step by hand
are in scripts/sdlc/docs/CI_TEST_COVERAGE.md.
6. Live-stack tiers (manual, plus the weekly probe schedule)
Section titled “6. Live-stack tiers (manual, plus the weekly probe schedule)”Each of these needs a deployed stack (or deploys its own) and has a written
procedure, and each but the first-time-user stories is mandatory for a release.
Several also run in CI in some form, on GitLab only, so a GitHub pull request covers
none of them: the dynamic API RBAC matrix is step 12 of the GitLab
integration_tests deployment (scripts/sdlc/codebuild_deployment.py shells out to
make api-test against the stack that job deploys); the ZAP DAST, tenant isolation
and deploy-variant rows run on the weekly variant-probe schedule, a scheduled pipeline of its own
separate from the nightly; the UX review runs unattended as the advisory
ai_ux_test job on merge requests that change src/ui/; and the first-time-user
stories run as advisory jobs on merge requests that change CHANGELOG.md and every
night against the review stack. Note when integration_tests runs: automatically
only in the scheduled pipelines; on a merge request to develop and on develop
itself it is a manual button, and on any other branch it does not run, so nothing
exercises a change before it merges unless someone presses that button. The rest of
the rows run only when a person asks.
| Tier | What only a live stack can prove | Command | Procedure |
|---|---|---|---|
API RBAC — dynamic (also runs in the GitLab integration_tests job) | that the deployed resolver enforces authorization per Cognito group and configuration-profile scope, not just that the code looks right | make api-test STACK_NAME=… | api-rbac-test |
| Cognito authorization behaviour | that the pre-token group-mapping trigger and client attribute permissions behave as the docs claim — they do not always | make live-auth-checks · make verify-idp-federation | live-auth-checks |
| UX review (browser) | that a person can actually complete each flow in the web UI, and how it feels doing so — functional pass/fail plus usability findings. One of the two tiers here that open a browser, beside the first-time-user stories below | make ux-test STACK_NAME=… | ux-test |
| First-time-user stories (browser) | that a person new to the product can do what the published documentation says they can, by following it literally — each story is one intent attempted as one persona and judged against what that persona’s role should be able to do, with every hesitation, dead end and gap between the docs and the screens recorded as it goes. Advisory in GitLab CI on merge requests and nightly; not a release tier | make ai-ux-test-stories-local STORY=… | ux-test-stories |
| Inbox end to end | that a message posted through idp_common.notifications (direct, Cognito group fan-out, idempotency key, dedupe key) is the one the deployed API lists, counts and marks read for its recipient, and that another user cannot reach it. Creates and removes a temporary Cognito group, two users and their Inbox items | make stacktest-inbox STACK_NAME=… | run-stack-tests |
| Review queues end to end | that a document routed under a profile with review queues on reaches the deployed queue, can be claimed, edited and completed through the API with a real Annotator token, and finishes with the final status object, an unchanged machine manifest and a final manifest. Creates and removes a temporary profile, Annotator, Admin and document, and lists that Annotator on the Default queue for the duration of the run (CHECK_EVENTS=1 adds a temporary EventBridge rule and SQS queue) | make stacktest-review-queues STACK_NAME=… | run-stack-tests |
| ZAP DAST (also on the weekly probe schedule, while it is active) | that the deployed HTTP surface has no exploitable finding | make stacktest-zap STACK_NAME=… | run-stack-tests |
| Tenant isolation (also on the weekly probe schedule, while it is active) | that a stack deployed with TenantIsolationMode=Strict keeps two tenants apart: each tenant’s users read their own document and are refused the other’s, strict-mode ingest refuses an upload outside every tenant’s prefixes, and the generated per-tenant access points refuse a cross-tenant upload. Creates and removes throwaway profiles, tenants, Cognito users, two IAM roles and an access point stack | make stacktest-tenancy (self-deploys a Strict stack) · make verify-tenant-isolation STACK_NAME=… (an existing Strict stack) | run-stack-tests |
| Deploy variants (also on the weekly probe schedule, while it is active) | that each hosting/parameter combination actually creates and serves | make stacktest-hosting-global · make stacktest-waf · make stacktest-hosting-private (VPC) · make stacktest-jobsapi (VPC) — list them with make stacktest-list | run-stack-tests |
| Template transforms | that a transformed template deploys and processes a real document — the only tier that can | make transform-deploy-test-headless · make transform-deploy-test-govcloud · both: make transform-deploy-test-all | transform-deploy-test |
| Seller Entitlement Service e2e | that the service deploys into a seller account, grants correctly, and refuses correctly | make stacktest-seller | run-stack-tests |
| Analytics interpreter network reach | what a session on the AWS-managed Bedrock AgentCore Code Interpreter the analytics agent runs Python on can reach: public DNS and HTTPS, a raw TCP connection, Amazon S3 in the session’s region and in another region, the regional STS endpoint, a PUT to a presigned URL (and whether the object arrives), and whether any AWS credential source is present. Compares each with the posture recorded in scripts/sdlc/sandbox_network_probe.py and prints a pass/fail table. Needs no stack; creates and deletes a throwaway bucket | make stacktest-sandbox-network | run-stack-tests |
| Terraform property parity | that a Terraform deployment of the release stores the same configuration as a CloudFormation deployment of it with the same parameters, every resource of the main and nested stacks read back through Cloud Control, beyond the explained differences in terraform/tools/parity_baseline.json. Read-only; needs both deployments | make terraform-parity CFN_STACK=… TF_STACK=… | release-validation (tier I) |
| In-place upgrade (X→Y) | that an existing customer stack survives update-stack without rollback and still works afterwards | make stacktest-upgrade (pointer) | test-upgrade |
Migration from genai-idp-terraform (also on the weekly Terraform-migration schedule, while it is active) | that a deployment of the separate awslabs/genai-idp-terraform implementation, at a pinned tag and seeded with documents, users, a configuration edit, a baseline and a test set, moves onto this repository’s module with every phase of idp-cli terraform migrate-from-genai-idp-terraform, keeping every object and item, and that afterwards both users sign in to the new deployment with their old passwords and keep their scope, a new document is processed by the new deployment only, and destroying the old deployment leaves the adopted resources standing. Deploys and destroys both deployments: measured end to end at 166 minutes, 76 of them waiting on DynamoDB’s two time-to-live intervals and 33 the adoption apply and writes result.json and summary.txt in its work directory | make stacktest-terraform-migration OLD_NAME=… NEW_NAME=… TERRAFORM_MODULE_URL=… | release-validation (tier J) |
The UX review, and recording it
Section titled “The UX review, and recording it”make ux-test STACK_NAME=… prepares a throwaway Cognito user and a session, then a
person or the assistant drives the flows in scripts/ux_flows.yaml in a real browser
and reports functional pass/fail per flow plus ranked usability findings. It is
deliberately not a script: a shell script can assert that a page returned 200, not
that the page made sense.
The review can also be recorded as a narrated, captioned video, so a finding can be shown to the team instead of re-demonstrated live:
make ux-record-deps # ffmpeg, ffprobe, boto3, Pillow./scripts/ux_recorder.py start --stack <STACK> --persona Admin --url-contains cloudfront \ --say "We start on the Test Studio sets tab."./scripts/ux_recorder.py mark "Open the annotation queue" --say "We open the queue …"./scripts/ux_recorder.py stop --say "That ends the review."AWS_PROFILE=default ./scripts/ux_recorder.py render --voice Ruth # --dry-run firstThe recorder attaches a second DevTools session to the tab being driven, captures
frames only when the screen changes, and re-paces the result for a viewer: idle
gaps clamped, paused stretches dropped, narration spoken by Amazon Polly, every
click drawn on the frame that was on screen when it happened. Output — review.mp4,
review.srt, segments.json, review.md — lands under gitignored
scratch/ux-recordings/, in a session directory that also holds the tab’s address
and title and a click log (events.jsonl, timeline.json) that labels clicks with
text from the page. A recording of a live stack shows real documents and
nothing is redacted: never commit the session directory or attach any of it to a
pull request. One exception is a finished product-demo video its author chooses
to publish: the author uploads that demo.mp4, and nothing else from the session, to
get its asset URL for the demo videos page. The MR UX test’s mp4 of the CI review
stack, which holds only shipped example data, is the other. What a click label can
hold is set out under Privacy in
ux-test.
Details:
scripts/README.md.
On a merge request the same review runs unattended. When a non-Draft MR to
develop changes src/ui/, the advisory ai_ux_test job tests the MR’s UI build in
a confined headless browser against the long-lived review stack idp-ux-nightly, and
appends its findings and a narrated video to the automated code-review comment for
that commit. It gates nothing. make ai-ux-test-local MR=<iid> runs the same test in
Docker as a dry run, and make ai-ux-browser starts only the confined browser, with
DevTools on 127.0.0.1:9222, for an interactive session on a machine with no local
Chrome. Contract:
ux-test-ci.
First-time-user stories
Section titled “First-time-user stories”scripts/ux_stories.toml
holds stories that say only what a person wants, each with the personas it runs as and
whether each persona’s role should be able to do it, be refused, or not be offered it
at all. A model that knows the product only through this documentation attempts each
story in a confined headless browser, searching and reading these pages through tools
of its own and noting every hesitation and dead end as it goes. Every tool call is
journalled, and the verdict — done, stuck, refused as the docs say, offered a control
the docs say the role lacks, and so on — is computed from that journal and the
persona’s expected access, never from the model’s prose, so one run of a story can be
compared with the next. Each story is told to change nothing on the shared review
stack, and a best-effort guard in front of the browser refuses the saves it
recognises: a story whose task ends in a save stops at that step and says so.
make ai-ux-test-stories-local STORY=<id> runs a story in Docker as a dry run
(MR=<iid> runs the stories that MR’s added CHANGELOG entries select, and
INTERACTIVE=1 lets you watch it live), make ai-ux-test-stories-pair starts the
confined browser and prints the command for a Claude session you start to play the
user, and make ai-ux-test-stories-list and
make ai-ux-test-stories-search Q="…" are the authoring loop. In CI, the advisory
ai_ux_test_stories job runs the stories a non-Draft MR’s added CHANGELOG entries
select and replies in the code-review thread, and every night the nightly set runs
and feeds a digest of regressions; none of it gates anything. Procedure:
ux-test-stories;
unattended contract:
ux-test-stories-ci.
7. Benchmarks
Section titled “7. Benchmarks”make benchmark-release VERSION=… PREV=… runs the release-vs-release A/B —
accuracy, completeness, cost, latency and confidence calibration against the
previous published release. The wider suite in benchmarks/ (a config × document-size
matrix with exact ground truth) is what produces the published
Configuration Guidance and is the gate for any
change that could move accuracy or cost.
Start at the Benchmarking Guide; read
benchmarks/matrices/METHODOLOGY.md before writing down a number, and the
run-benchmarks
procedure before running one.
Where results are recorded
Section titled “Where results are recorded”A test that leaves no record cannot be cited later, so three of the layers above write one:
| Record | What it holds |
|---|---|
| Release Validation Records | one file per release, never overwritten: every tier that needs a live stack, its verdict, and what it found |
security/test-results/<version>/ | redacted per-test snapshots of SRT, ZAP DAST and RBAC static + dynamic |
| Release Benchmark Audit Trail | the release-vs-release A/B, with n and spread on every claim |
Everything else — raw logs, probe output, UX recordings — stays in gitignored
scratch/. Records are public-safe by construction: account ids, VPC/subnet/security-group
ids, API hostnames, pool ids, stack physical ids and local paths are replaced with
placeholders.
Running the whole battery
Section titled “Running the whole battery”One release validation exercises every layer on this page in a fixed order, on real
stacks, and writes the three records above. It is a two-day job, driven by the
release-validation
procedure — the umbrella over every per-tier procedure linked here. The per-tier
make targets stay runnable on their own, which is how you validate a single change
without validating a release.