Testing

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

Testing

Every test layer and tier in this repository: what it proves, how to run it, whether CI runs it for you, and where its results are recorded. This is a map of the tiers, not an index of individual test functions — there are thousands of those across hundreds of test modules, and one added inside a suite that already runs needs no change here. make test-list enumerates the suites themselves, and the section on what make test does not run is the one place the suite-level exceptions are written down. If you are looking for what a given release was actually validated against, that is the Release Validation Records; this page describes the methods, not any particular run of them.

The organising fact: most of what protects this repo runs on every pull request, but the tiers that need a deployed stack cannot. Roughly a dozen methods below run only when a person asks for them, which is why each one has a written procedure and a record of its last run.

The layers at a glance

Layer	Needs AWS?	Runs in CI?	Entry point
1. Offline test suites	no	✅ both CIs	make test
2. Static gates	no	✅ both CIs	make lint-cicd · make typecheck-pr
3. Web UI unit tests	no	✅ both CIs	make ui-test
4. Security scanning	no	✅ both CIs	make srt-scan · make dep-audit
5. Integration smoke suite	yes	⚠️ GitLab only	pipeline integration_tests
6. Live-stack tiers	yes	❌ manual	see the table
7. Benchmarks	yes	❌ manual	make benchmark-release

make test and make lint are the two commands to run before opening a pull request. make all is both.

1. Offline test suites

make test discovers and runs every non-integration suite in the repo — the idp_common library, idp_cli, idp_sdk, idp_feature_sdk, the feature platform, the per-Lambda suites, the config library, and the repo’s own tooling tests under scripts/, with the handful of documented exceptions below. make test-list prints the discovered roots without running them, which is the honest answer to “is my new suite actually being run?”.

make test                                   # everything, auto-discovered
make test-list                              # which roots were discovered
cd lib/idp_common_pkg && make test-unit     # one root, isolated
make test-integration-all                   # only the integration-marked tests (needs AWS)

CI runs the same suites split across two targets — make test-cicd -C lib/idp_common_pkg and make test-packages-cicd — so a suite that exists but is wired into neither is invisible to CI even though make test runs it locally.

There is no standing failure set — Expected standing failures: 0 on a correctly installed tree, and the enumerated list of accepted failures in full-test-battery is empty. So treat any failure as a real regression until proven otherwise. scripts/tests/test_standing_failure_baseline.py holds that claim, this page and the two skills that repeat it to the same number, so they cannot drift apart again.

Most surprising failures are still a stale virtualenv missing the pinned [test] extras — but do not expect a broken install to announce itself as an ImportError. Several Lambdas catch a missing idp_common on purpose and degrade (feature-platform/main-stack-extensions/lambdas/apply_feature_config_preset/index.py logs at ERROR and applies a config preset without recording a revision), so a suite that exercises the non-degraded path fails on a bare assertion instead. Two tests in test_apply_feature_config_preset.py (test_remove_hands_the_pipeline_back_to_default_then_deletes and test_remove_keeps_an_active_profile_when_there_is_no_default_to_fall_back_to) were misread as a standing failure of this repo for exactly that reason. With idp_common unimportable that file reports 2 failed, 18 passed; with PYTHONPATH=<checkout>/lib/idp_common_pkg exported it reports 20 passed, on the same interpreter and the same commit. Measured identically under Python 3.12 and 3.13, so it is not a version incompatibility. Check that python3 -c "import idp_common; print(idp_common.__file__)" resolves inside your own checkout before reading anything else — an editable install can silently point at a different, or deleted, checkout. The diagnosis order, the per-suite expected totals, and how to prove a failure is inherited are in the full-test-battery procedure. Conventions for writing tests — pytest markers, moto, conftest layout — are in testing-qa.

Suites make test does not run

Auto-discovery means a suite cannot be forgotten, not that every suite is run. scripts/run_all_tests.py refuses to run at all when it finds a directory holding a test_*.py that is in neither of its two registries — the roots it runs, and the roots it excludes with a written reason — so tests in a new location cannot be silently skipped. An exclusion covers only the directory named: nesting under an excluded directory used to inherit the exclusion, which meant excluding scripts quietly accepted every future test directory beneath it, so each excluded directory is now listed on its own. That check backs make test, which runs in neither CI, so scripts/tests/test_testing_doc.py re-derives it on every pull request, where pytest scripts/tests does run.

These are the directories in the excluded registry. They are listed because a suite that exists and never runs is otherwise indistinguishable from one that passes:

Not run by make test	Why
scripts	scripts/test_api_rbac.py is the live RBAC harness driven by make api-test against a deployed stack (layer 6), not a pytest suite; collecting it picks up its test_email() helper as a test
src/lambda/ocr_benchmark_deployer	test_local.py needs huggingface_hub, which is not a test dependency
nested/bedrockkb/src/s3_vectors_manager	test_handler.py imports cfnresponse, which exists only in the Lambda runtime
nested/bedrockkb/src/s3_vectors_manager/tests	Named separately now that an exclusion no longer covers what is nested under it. Not skipped in practice — make test-packages-cicd runs it directly, in both CI systems, so CI runs more than make test does
samples/lambda-hook-inference/GENAIIDP-chandra-ocr-hook	test_local.py is a manual local-run script and collects zero pytest tests (measured)
lib/idp_sdk/idp_sdk/_core	source, not tests: test_studio_processor.py is the Test Studio processor module, which the test_ prefix makes look like a suite

Adding an exclusion, or lifting one of these, fails that guard until this table and the registry agree — it is checked in both directions, so a row that outlives the exclusion it describes fails too.

2. Static gates (lint, types, and hand-written scanners)

make lint is the local gate; make lint-cicd is the same set in check-only mode and is what both CIs run. Several members of the set are not linters at all but hand-written scanners for classes of defect that have each shipped at least once:

Gate	What it catches
make ruff-lint · make format	Python style and formatting (88 cols, Python 3.12)
make cfn-lint	any file declaring AWSTemplateFormatVersion — discovered by content, so a new template cannot escape it. Fails on errors only
make check-arn-partitions	hardcoded arn:aws: / amazonaws.com instead of ${AWS::Partition} / ${AWS::URLSuffix} — GovCloud compatibility
make check-filtered-scans	DynamoDB Scan with a filter expression that cannot see all matches
make check-data-plane-tags	the idp:plane=data tag on the Lambdas that must carry it
make validate-buildspec	malformed CodeBuild buildspecs — otherwise a deploy-time failure
make codegen-check	generated GraphQL types drifting from the schema
make typecheck · make typecheck-pr	basedpyright; CI checks only files the PR changed
make api-test-static	an API operation added without authorization, and drift between the dispatcher’s generated required-groups manifest and scripts/api_rbac_expectations.yaml — see layer 6 for the live half
python3 scripts/check_first_party_deps.py	a first-party package installed by bare name, which on public PyPI is somebody else’s code
python3 scripts/sdlc/validate_service_role_permissions.py	the CloudFormation service role missing a permission the templates need

Two of these gates guard the gates themselves: scripts/tests/test_ci_gate_parity.py fails if a gate runs in one CI and not the other, or if lint-cicd becomes weaker than local make lint; and scripts/tests/test_nested_stack_parameters.py checks parent-to-nested stack parameter wiring that cfn-lint’s own rule cannot see. Both exist because every parity gap they cover was originally found by hand, months late.

Whether any of this actually blocks a merge

Parity means both CIs run a gate. Whether a red gate can stop a merge is a repository setting, and today it does not: develop has no branch protection, so every gate in this table is advisory — a pull request can be merged with all checks red, and because the GitHub workflows are pull_request-only, a direct push to develop runs none of them.

make check-branch-protection    # reads the live setting via the GitHub API

The check derives the expected required-check list by parsing .github/workflows/*.yml for the job names GitHub turns into status-check contexts, rather than from a hardcoded list that would drift on the next rename. It then asserts protection is enabled, that every context a PR produces is required, that stale approvals are dismissed, that force-push and deletion are blocked, that an approving review is required, and that enforce_admins is on — without it an administrator can push straight past everything else. It also names the contexts that must stay advisory: the docs and dependency-manifest workflows are path-filtered, and Test Results is a check run an action creates behind an if:, so none of them reports on every PR and requiring one would leave a check pending forever and block every merge.

Three details are worth knowing about what it reads. All eight shared gates are steps inside one job, so they collapse to a single requireable context and share a single red mark — a required-check failure does not say which of the eight failed. It reads both enforcement mechanisms, classic branch protection and rulesets, because a branch can be fully governed by a ruleset while the classic endpoint reports nothing. And it distinguishes “not protected” from “cannot see”: the classic endpoint needs repository admin and answers 404 without it, so the tool cross-checks GET /repos/{slug}/branches/{branch}, which carries a protected boolean and is readable with plain pull access. --json therefore reports protected: null — not false — when the state genuinely could not be determined.

It is opt-in and blocks nothing: it needs network access and a token (pull access is enough to reach a verified answer about whether the branch is protected and to compare the required-check list, because GET /repos/{slug}/branches/{branch} carries a nested protection.required_status_checks object at that scope; administration:read is what the other five assertions — approvals, stale-review dismissal, force-push and deletion blocks, enforce_admins — need, and a run without it reports those five as unread, not as satisfied), and it reports “not protected” until issue #933 is closed, since enabling protection needs repository admin. With no token or no network it exits 0 with an explanation. Once #933 closes it should become a required, blocking check, run with --fail-on-skip. Its own parsing and assertion logic is covered offline by scripts/tests/test_check_branch_protection.py.

3. Web UI unit tests

make ui-test        # Vitest, jsdom — no browser
make ui-build       # lint + typecheck + production Vite build

Both CIs run the Vitest suite. Nothing in this layer opens a browser — that is layer 6’s UX review, and it is the only tier in the repo that does.

4. Security scanning (SAST and SCA)

make srt-scan       # Sample Security Review Tool: SAST over the checkout
make dep-audit      # every pinned Python + Node dependency against OSV (fails on HIGH+)

Both run on every pull request in both CIs, and neither needs AWS. They cover different things and one does not imply the other: SRT’s syft stage builds an SBOM (inventory only, no vulnerability matching), so dependency CVEs are dep-audit’s job alone.

Findings are triaged in place rather than waved through: bandit findings take a line-scoped # nosec <ID> - <reason>, unreachable advisories go in scripts/security/dep_audit_allowlist.json with a justification, and security-matrix/Checkov findings go in scripts/srt/issues.json. Procedures: srt-security-scan.

Published, redacted snapshots of the four security tests per release live in security/test-results/<version>/, produced by make security-results.

5. Integration smoke suite (CI only)

The GitLab pipeline’s integration_tests job deploys a stack and drives fourteen numbered steps through it — default config, BDA mode, rule validation, concurrent batch processing, Test Studio evaluation, agentic extraction on a large table, single- and multi-document discovery, test comparison, API RBAC, IAM permissions boundary, and pipeline hooks. It needs AWS credentials, so it is GitLab-only: a change merged through a GitHub pull request has not run it.

Per-step detail, what each step asserts, and how to reproduce a single step by hand are in scripts/sdlc/docs/CI_TEST_COVERAGE.md.

6. Live-stack tiers (manual)

None of these run in CI. Each needs a deployed stack (or deploys its own), each has a written procedure, and each is mandatory for a release.

Tier	What only a live stack can prove	Command	Procedure
API RBAC — dynamic	that the deployed resolver enforces authorization per Cognito group and configuration-profile scope, not just that the code looks right	make api-test STACK_NAME=…	api-rbac-test
Cognito authorization behaviour	that the pre-token group-mapping trigger and client attribute permissions behave as the docs claim — they do not always	make live-auth-checks · make verify-idp-federation	live-auth-checks
UX review (browser)	that a person can actually complete each flow in the web UI, and how it feels doing so — functional pass/fail plus usability findings. The only tier here that opens a browser	make ux-test STACK_NAME=…	ux-test
ZAP DAST	that the deployed HTTP surface has no exploitable finding	make stacktest-zap STACK_NAME=…	run-stack-tests
Deploy variants	that each hosting/parameter combination actually creates and serves	make stacktest-hosting-global · make stacktest-waf · make stacktest-hosting-private (VPC) · make stacktest-jobsapi (VPC) — list them with make stacktest-list	run-stack-tests
Template transforms	that a transformed template deploys and processes a real document — the only tier that can	make transform-deploy-test-headless · make transform-deploy-test-govcloud · both: make transform-deploy-test-all	transform-deploy-test
Seller Entitlement Service e2e	that the service deploys into a seller account, grants correctly, and refuses correctly	make stacktest-seller	run-stack-tests
In-place upgrade (X→Y)	that an existing customer stack survives update-stack without rollback and still works afterwards	make stacktest-upgrade (pointer)	test-upgrade

The UX review, and recording it

make ux-test STACK_NAME=… prepares a throwaway Cognito user and a session, then a person or the assistant drives the flows in scripts/ux_flows.yaml in a real browser and reports functional pass/fail per flow plus ranked usability findings. It is deliberately not a script: a shell script can assert that a page returned 200, not that the page made sense.

The review can also be recorded as a narrated, captioned video, so a finding can be shown to the team instead of re-demonstrated live:

make ux-record-deps                                    # ffmpeg, ffprobe, boto3, Pillow
./scripts/ux_recorder.py start --stack <STACK> --persona Admin --url-contains cloudfront \
    --say "We start on the Test Studio sets tab."
./scripts/ux_recorder.py mark "Open the annotation queue" --say "We open the queue …"
./scripts/ux_recorder.py stop --say "That ends the review."
AWS_PROFILE=default ./scripts/ux_recorder.py render --voice Ruth   # --dry-run first

The recorder attaches a second DevTools session to the tab being driven, captures frames only when the screen changes, and re-paces the result for a viewer: idle gaps clamped, paused stretches dropped, narration spoken by Amazon Polly, every click drawn on the frame that was on screen when it happened. Output — review.mp4, review.srt, segments.json, review.md — lands under gitignored scratch/ux-recordings/. A recording of a live stack shows real documents and nothing is redacted: never commit one or attach it to a pull request. Details: scripts/README.md.

7. Benchmarks

make benchmark-release VERSION=… PREV=… runs the release-vs-release A/B — accuracy, completeness, cost, latency and confidence calibration against the previous published release. The wider suite in benchmarks/ (a config × document-size matrix with exact ground truth) is what produces the published Configuration Guidance and is the gate for any change that could move accuracy or cost.

Start at the Benchmarking Guide; read benchmarks/matrices/METHODOLOGY.md before writing down a number, and the run-benchmarks procedure before running one.

Where results are recorded

A test that leaves no record cannot be cited later, so three of the layers above write one:

Record	What it holds
Release Validation Records	one file per release, never overwritten: every tier that needs a live stack, its verdict, and what it found
security/test-results/<version>/	redacted per-test snapshots of SRT, ZAP DAST and RBAC static + dynamic
Release Benchmark Audit Trail	the release-vs-release A/B, with n and spread on every claim

Everything else — raw logs, probe output, UX recordings — stays in gitignored scratch/. Records are public-safe by construction: account ids, VPC/subnet/security-group ids, API hostnames, pool ids, stack physical ids and local paths are replaced with placeholders.

Running the whole battery

One release validation exercises every layer on this page in a fixed order, on real stacks, and writes the three records above. It is a two-day job, driven by the release-validation procedure — the umbrella over every per-tier procedure linked here. The per-tier make targets stay runnable on their own, which is how you validate a single change without validating a release.