Versioning and Compatibility
CATS follows Semantic Versioning 2.0.0, binding from 1.0.0. This page is the contract: it names every surface the promise covers, states the rule each surface follows, and lists what is not covered.
The governing rule: only the surfaces named on this page carry compatibility guarantees; anything else CATS exposes, documented or not, may change in any release.
Two definitions apply everywhere below:
- A breaking change to a covered surface requires a major version. Rename, removal, a change of type or meaning, and a change of default are breaking.
- An additive change to a covered surface is a minor version: a new export, a new optional field, a new command or flag, a new enum member, a new key in a document. Code written against the previous minor keeps working.
Product stance
CATS records every test in the session in report.json and junit.xml. The @cats
decorator is optional metadata: a test that attests to no standard is a first-class case,
not a degraded one. Compliance outputs, the OSCAL findings and the Security Hub findings,
act only on tests that carry a standard.
The compatibility contract
| Surface | Covered | Breaking change means |
|---|---|---|
Python exports: every name in cats.__all__ |
Yes | Rename, removal, signature change, behavior change of documented semantics |
Dynamic namespaces: account.<target>, account.role.<name>, and their reserved-name rules |
Yes | Adding a reserved name that a config could have used |
The @cats decorator: its three keywords and the custom-marker channel |
Yes | Any new keyword; any change to the three |
| Pytest plugin surface: fixture names and scheme, marker names, command-line options | Yes | Any change to the naming derivation; rename or removal of a fixture, marker or option |
cats.yaml |
Yes | Field rename or removal, type narrowing, constraint tightening, semantic change, new required field, default change |
| CLI: commands, arguments, flags, defaults, exit codes, streams | Yes | Rename or removal; changed default; changed exit-code or stream semantics |
report.json and the report file naming scheme |
Yes | Field removal or type change; renaming a file; a change to schema_version |
Security Hub: the finding Id scheme, the cats/* ProductFields keys, the status mapping |
Yes | Any change to the Id scheme; rename or removal of a key; a change to the mapping |
Exceptions: the CatsError root rule |
Yes | A deliberate raise that is not a CatsError; moving an error between the three caller-facing classes |
| CloudFormation outputs: the output key scheme | Yes | Key rename or removal; a change to the {target} suffix rule |
| Everything in Not covered | No | May change in any release |
Python exports
The covered Python API is exactly cats.__all__. It is pinned by
tests/unit_tests/test_contract_surface.py; a change to that test is a contract change.
At 1.0.0 the list is:
| Group | Names |
|---|---|
| Objects | AccountUnderTest, NetworkHandler, CatsReporter, CatsTestPack, StandardMetadata, CatsConfig (the loaded cats.yaml, as the pytest_cats_provides hook receives it: its fields are the configuration keys and are covered; its methods are internal) |
| Result types | CatsNetworkResponse, CatsNetworkResult, ShellResult |
| Exceptions | CatsError, CatsUsageError, CatsConfigError, CatsInfrastructureError, CatsVersionMismatchError, CatsSSMError, CatsLambdaInvocationError |
| Decorator | cats, cats_requires |
| Helpers | get_account_fixture_names, get_accounts_by_tag, GetAccountFixture |
| Metadata | __version__ |
Covered, for each name: the documented members, every attribute and method whose name
does not start with _, the keyword names of every parameter, and the positional order of
every positional parameter. Anything starting with _ is internal.
AccountUnderTest(name)is the constructor: one positional argument, the account keyname fromcats.yaml. An unknown name raisesCatsConfigError.NetworkHandleris constructed byAccountUnderTestand reached asaccount.<target>. Its__init__is not covered; do not construct it.CatsNetworkResponsemembers andShellResultfields may be added in a minor; removing or renaming one is major.
Dynamic namespaces
AccountUnderTest.<target> resolves the network targets declared in cats.yaml, so every
attribute of AccountUnderTest is a name a config cannot use. The rule that keeps the two
apart:
- The reserved single-word attributes of
AccountUnderTestare frozen at exactlyname,partition,session,role,tags. A target with one of these names is rejected when the config loads. - Every attribute added to
AccountUnderTestafter 1.0.0 contains an underscore. Target names cannot contain one, so no future attribute can collide with a target. account.role(SessionHandler) has one public name,default. A role's name (its key inadditional_roles) may not start with_; everything onSessionHandlerthat is notdefaultdoes.- Case-only collisions are rejected: two account keynames, or two target names in one account,
that differ only in case are a config error. Two role names in one account must differ by
more than case and underscores: IAM role names are case-insensitive, and the CloudFormation
logical ID drops
_.
The @cats decorator
- The keyword set is frozen at
standard_name,control_ids,control_group.@catsnever gains a keyword. standard_nameis optional.@cats(team="platform")on a test that attests to nothing is first-class: the marker is recorded, and no standard is attached.- Every other keyword is a custom marker. It must be declared in
reporting.custom_markersincats.yaml; an undeclared keyword raisesCatsUsageErrorwhen the test module is imported. - Future per-test behavior never arrives as a
@catskeyword. It is configured incats.yamlagainst a user-declared marker, or it arrives as a newcats_*pytest marker.@cats_requires(*names)is the first: a sibling decorator that applies onecats_requires(name=...)mark per requirement, read bytest_selectionincats.yaml. Its positional arguments are free-form requirement names; it takes no keyword.
Pytest plugin surface
- Account fixtures are named
<keyname>_account, one per account incats.yaml. No CATS fixture added after 1.0.0 ends in_account. - CATS's own fixtures are
cats_reportandget_account_fixture. - Markers. Every marker CATS registers is named
cats_*.@catsemitscats_standard(name, control_ids, control_group)and@cats_requiresemitscats_requires(name); select on either with-m "cats_standard(name='SOC2')"or-m "cats_requires(name='container-deployment')". A custom marker declared inreporting.custom_markersmay not start withcats_.test_selection.excludednames the two engine markers as they are written on the test,standard_nameandcats_requires, so it reserves no custom-marker name. Every junit property CATS writes is prefixedcats:(cats:standard,cats:account,cats:tag:<key>,cats:xfail,cats:description, andcats:run_id,cats:role_session_name,cats:caller_identity_arn,cats:git_commit,cats:git_dirty,cats:config_sha256andcats:test_packon the testsuite); a custom marker's or metadata key's property keeps the user's name. Anot_runtestcase's<skipped>element hastype="cats.not_run". - Hooks a conftest may implement:
pytest_cats_standards,pytest_cats_test_pack,pytest_cats_provides(cats_config). Parameter names are part of the contract (pluggy matches implementers by name). - Command-line options the plugin adds to pytest:
--cats-report-dir <dir>.
cats.yaml
The schema is covered. The rules:
- Breaking (major): field rename or removal, type narrowing, constraint tightening, semantic change, a new required field, a change of default.
- Additive (minor): a new optional field, type widening, constraint relaxation.
- Keys are case-sensitive, at every level including the top-level sections.
aws_regionis required unless theAWS_REGIONenvironment variable is set. CATS never picks a Region for you.enabled_network_handlersneeds at least one handler wherever it appears. A declared target exists to be probed; an account with nothing to probe omitsaccount_network_under_test.subnet_idis an input-only spelling for a single subnet. It is folded intosubnet_idson load, and a dumped config emitssubnet_idsalone.
CLI
Commands, arguments, flags, defaults, exit codes and streams are covered. The command reference is CLI.
Exit codes, defined once for every command:
| Code | Meaning |
|---|---|
0 |
Success. For cats infra status, "not deployed" is a successful answer |
1 |
The operation ran and failed: a stack failed to deploy, an account could not be read, an AWS call failed |
2 |
Usage or configuration error: unknown flag, missing required option, invalid cats.yaml, a --config path that is not a file |
130 |
Interrupted (Ctrl+C) |
A batch that mixes usage and operational failures reports 2.
The one exception is the test run itself: once cats test has handed over to pytest, it
returns pytest's exit code unchanged (0 all passed, 1 tests failed, 2 interrupted,
3 internal error, 4 usage error, 5 no tests collected), or 128 + N when the pytest
process was ended by signal N before it could exit (143 for SIGTERM, the usual result
of a cancelled run; CLI reference).
Streams: results to stdout, diagnostics to stderr. A result is what the command
exists to produce: JSON from cats run, a template from cats infra show-template, a
table from cats infra status. Failure text, warnings and progress are diagnostics.
The config option is spelled --config on every command that takes one, with -c as
the short form everywhere except cats test, which passes unknown arguments to pytest, and
pytest owns -c.
report.json and the report files
report.jsonhas three top-level keys:schema_version,runandtests.schema_versionis1at 1.0.0 and increments only with a breaking change to the document. Additive fields are minor and do not move it. The corollary for a consumer: a reader must tolerate keys it does not know (no strict schema, no closed deserializer), because new keys arrive in minor releases.- Entry identity is the pytest node id (
tests/test_x.py::test_y[param]) relative to pytest's rootdir.cats initpins the rootdir with an ini file at the project root: a[tool.pytest.ini_options]section in an existingpyproject.toml, else apytest.ini. The node id is the key of every entry, the sort order of the file, and the seed of every OSCAL and Security Hub identifier. - Results are
passed,failed,skipped,xpassed,xfailed,errorandnot_run. The six pytest outcomes are frozen.not_runis a testtest_selectionheld back, recorded with its reason; it counts incounts.totalandcounts.not_run, andrun.selectioncarriesprovided,config_excludedandunmatched_exclusionsbeside the command-line selectors. A new result value would be major. - Timestamps. Every timestamp CATS owns (
started_at,finished_at,recorded_at, the OSCAL metadata) is RFC 3339, UTC, microsecond precision,Zsuffix:2026-09-24T10:53:17.000000Z.run_idisstarted_atto the second in basic form,20260924T105317Z; a second session starting in the same second against the same directory gets a-2(-3, ...) suffix so no file is overwritten. - File names. Four files, each with a fixed name that always holds the newest run and a
timestamped name that is written once:
report.json/report-<run_id>.json,assessment-results.json/assessment-results-<run_id>.json,junit.xml/junit-<run_id>.xml,securityhub-findings.json/securityhub-findings-<run_id>.json.reporting.latest_filesandreporting.timestamped_fileschoose which are written.
Security Hub
Covered, because a finding Id is sticky for 90 days in the customer's hub:
- The finding
Idscheme:cats/{cfn_stack_prefix}{keyname}/{aws_region}/{uuid5(nodeid)}. - The fixed
ProductFieldskey names:cats/version,cats/run_id,cats/nodeid,cats/account_name,cats/aws_region,cats/result,cats/accounts_in_test,cats/error_message,cats/reason, and the prefixescats/marker/<name>andcats/metadata/<key>. Rename or removal is major; a new key is minor. - The result mapping:
pytest result Compliance.StatusRecordStatefailed FAILEDACTIVEerror NOT_AVAILABLEACTIVExfailed FAILEDACTIVEpassed, xpassed PASSEDARCHIVEDskipped NOT_AVAILABLEARCHIVEDnot_run not imported —
- A
not_runtest is never imported: it produced no control state. Its count is in the summary line and insecurityhub-findings.jsonasnot_run. securityhub-findings.jsoncarriesschema_versionand follows thereport.jsonrule: additive is minor,schema_versionmoves only for a breaking change.securityhub.only_with_standarddefaults totrue: only tests carrying a standard become findings, per the product stance above.
Not covered: the Types strings and the Severity mapping.
Exceptions
Everything CATS raises deliberately is a CatsError. Callers see three classes for their own
mistakes and their environment:
| Class | Raised for |
|---|---|
CatsUsageError |
A bad argument: a URL where a hostname was expected, an invalid port, an undeclared @cats keyword |
CatsConfigError |
Your configuration: an unknown account name, a feature not enabled, an invalid cats.yaml (pydantic's detail is embedded in the message) |
CatsInfrastructureError |
AWS infrastructure missing or failing: no stack, another release's stack (CatsVersionMismatchError), and credential or endpoint failures at the session boundary |
CatsTestPack is a pydantic model; constructing it with invalid values raises pydantic's
ValidationError, not a CatsError.
CloudFormation outputs
The output keys of a CATS stack are covered so that tooling which reads stack outputs can rely on them. CATS itself reads outputs only from a stack of its own release (see below), so the scheme protects external readers, not CATS.
Three fixed keys per stack:
| Key | Value |
|---|---|
TestRoleArn |
ARN of the test role. Read by the deploy readiness gate, which assumes it to prove it usable; cats test rebuilds the name from the configuration instead |
ResponseBucketName |
The S3 bucket UDP receipts are written to |
UdpReceiptToken |
The token a UDP datagram carries for the listener to record a receipt |
Six keys per network target, each suffixed with the target name exactly as written in
cats.yaml ({Key}{target}, so target private gives Ec2DnsNameprivate):
| Key | Value |
|---|---|
Ec2DnsName{target} |
Private DNS name of the EC2 listener |
Ec2InstanceId{target} |
Instance ID of the EC2 listener |
VpcId{target} |
The resolved VPC ID |
SubnetIds{target} |
The resolved subnet IDs, comma-joined |
EnabledHandlers{target} |
The deployed handlers, comma-joined (ec2 or ec2,lambda) |
ProbeLambdaName{target} |
The probe function's name. Present only when lambda is an enabled handler |
Template internals behind the outputs are not covered.
Every release recreates the stacks
cats infra deploy tags every stack cats:version=<installed release>. A stack deployed by
any other release, patch releases included, is refused at test time (cats test, cats run,
the MCP tools) and at deploy time, with the same instruction:
cats infra destroy <account> && cats infra deploy <account>. A stack update never replaces
deployed code, so recreation is the only path across a release boundary.
This is by design and is not a breaking change of any release. CATS_SKIP_VERSION_CHECK=1
downgrades the test-time refusal to a warning for diagnosis; deploy has no bypass.
Not covered
These may change in any release:
- MCP tool names, arguments and schemas.
- The JSON body
cats runemits. - The OSCAL Assessment Results document shape.
- The hookspecs
pytest_cats_standardsandpytest_cats_test_pack. TheCatsTestPackclass they return is covered as an export; the hooks are not. reasontext inreport.jsonentries and onCatsNetworkResult: human-readable, no stable format.- Security Hub
Typesstrings and theSeveritymapping. - CloudFormation template internals: resource shapes, logical IDs, instance user data, SSM document content, Lambda source.
- Stack names and role names.
- Generated types from
cats generate-types: regenerate after every upgrade. - Internal modules: anything not in
cats.__all__, and anything whose name starts with_.
Deprecation cycle
- Breaking changes to any covered surface require a major version.
- Deprecations are felt, not just filed: a runtime
DeprecationWarningat the point of use, plus a documented migration path in the changelog. Deprecated config fields warn at config load; deprecated CLI flags warn on use. - A deprecated interface survives at least one minor release with its warning before removal, and removal happens only in a major.
- A changelog entry alone does not constitute a deprecation.
Python version support
- CATS supports every CPython version that has not reached upstream end-of-life, subject to the code's genuine floor. At 1.0.0 that is a 3.11 floor, and the supported set is 3.11, 3.12, 3.13 and 3.14, each in CI.
- A new CPython version is added to CI and declared supported within three months of its GA.
- A Python version is dropped in a minor release, no earlier than its upstream EOL date, announced in the changelog. Dropping a Python version is not a breaking change and does not require a major.