Helpers
Every name below is exported from cats; import it as from cats import .... The module
paths shown by the renderer are internal and may move.
Results and helpers
Helper functions and result types for use in tests, alongside the fixtures.
get_account_fixture_names and get_accounts_by_tag drive test
parameterization across accounts.
CatsNetworkResponse
Bases: StrEnum
Outcome of a network call, classified by what happened on the wire.
Each value is a precise claim about what the probing endpoint observed. Which control produced it is interpretation, and belongs in the test's docstring.
- SUCCESS: Connection established. For HTTP/HTTPS, any HTTP response (including 4xx/5xx and 3xx, which are not followed) is SUCCESS - the call arrived. The status code is on CatsNetworkResult.http_status for tests that care.
- REFUSED: RST received (ECONNREFUSED or ECONNRESET) - the network path is OPEN but nothing accepted the connection. For HTTP, a listener that accepted the connection and closed it without any response is also REFUSED. On a blocked-traffic test this means the control is NOT working. Indistinguishable from a Network Firewall REJECT.
- TIMEOUT: The TCP connect got no answer within the budget - consistent with a silent drop. This is the expected outcome for SG/NACL/Network Firewall DROP/ route-table isolation tests, but the endpoint cannot distinguish policy enforcement from transient loss: all silent drops look the same.
- DNS_FAILURE: Name resolution failed - the resolver answered NXDOMAIN/NODATA or was unreachable. What Route 53 Resolver DNS Firewall blocks produce, and also what a typo produces. No packet was ever sent to the target.
- TLS_FAILURE: TCP connected, the TLS handshake failed - untrusted or mismatched certificate, a plaintext listener on the port, or a protocol mismatch. What TLS inspection with an untrusted CA produces. The OpenSSL text is in reason.
- FAILURE: The path could not be proven open or blocked. Either a connect that succeeded with no HTTP reply inside the budget (a hung backend, NOT a silent drop; reason starts "connected; no HTTP reply"), or an unclassified network error (ENETUNREACH, EHOSTUNREACH, EADDRNOTAVAIL, ...; the errno is in reason). Never an expected value.
- SENT: UDP datagram left the host (connectionless - use receive_s3 to confirm arrival). REFUSED and TIMEOUT are unreachable for UDP.
CatsNetworkResult
dataclass
Outcome of a network call: a CatsNetworkResponse plus the evidence behind it.
Equality and hashing are category-only, against another result or a bare
CatsNetworkResponse: result == CatsNetworkResponse.TIMEOUT is the assertion
to write. reason, http_status, and correlation_id are diagnostics: they
are excluded from equality so error strings never become load-bearing, and
they are printed by the repr so a failing assertion is self-diagnosing.
Comparing to a plain string is refused (result == "TIMEOUT" is False): the
enum is the contract.
Attributes:
| Name | Type | Description |
|---|---|---|
response |
CatsNetworkResponse
|
The classified outcome. |
reason |
str | None
|
str(exception) from the probe for non-SUCCESS/SENT outcomes. |
http_status |
int | None
|
The HTTP status code, only for HTTP/HTTPS SUCCESS. |
correlation_id |
str | None
|
The UDP receipt ID that was sent or attempted, only for UDP. |
value
property
The response category as a string, for CLI/JSON consumers.
ShellResult
pydantic-model
Bases: BaseModel
Result of NetworkHandler.shell().
stdout and stderr are stripped of trailing whitespace. stdout_truncated and
stderr_truncated are set when SSM returned exactly its maximum for that stream (24,000
characters of stdout, 8,000 of stderr, before stripping): SSM keeps the first N characters
and gives no other signal, so an output of exactly N characters reads as truncated and the
remainder is not recoverable through CATS. Write large output to S3 from the command instead.
Show JSON schema:
{
"description": "Result of NetworkHandler.shell().\n\n``stdout`` and ``stderr`` are stripped of trailing whitespace. ``stdout_truncated`` and\n``stderr_truncated`` are set when SSM returned exactly its maximum for that stream (24,000\ncharacters of stdout, 8,000 of stderr, before stripping): SSM keeps the first N characters\nand gives no other signal, so an output of exactly N characters reads as truncated and the\nremainder is not recoverable through CATS. Write large output to S3 from the command instead.",
"properties": {
"stdout": {
"title": "Stdout",
"type": "string"
},
"stderr": {
"title": "Stderr",
"type": "string"
},
"exit_code": {
"title": "Exit Code",
"type": "integer"
},
"duration_ms": {
"title": "Duration Ms",
"type": "integer"
},
"stdout_truncated": {
"title": "Stdout Truncated",
"type": "boolean"
},
"stderr_truncated": {
"title": "Stderr Truncated",
"type": "boolean"
}
},
"required": [
"stdout",
"stderr",
"exit_code",
"duration_ms",
"stdout_truncated",
"stderr_truncated"
],
"title": "ShellResult",
"type": "object"
}
Fields:
-
stdout(str) -
stderr(str) -
exit_code(int) -
duration_ms(int) -
stdout_truncated(bool) -
stderr_truncated(bool)
get_account_fixture_names()
Get fixture names for all configured accounts.
get_accounts_by_tag(key, value=None)
Return fixture names for accounts with this tag.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str
|
Tag key to match |
required |
value
|
str | None
|
If provided, tag must have this value. If None, tag just needs to exist. |
None
|
Returns:
| Type | Description |
|---|---|
list[str]
|
List of fixture names (e.g., ["prod_account", "dev_account"]) |
Reporting
CatsReporter
The evidence a test adds to its own report entry.
One instance per test, provided by the function-scoped cats_report fixture (and by
get_account_fixture, which records the accounts it resolves). add_account puts an
account into accounts_in_test when the test reached it some way other than a fixture
argument; add_report_metadata attaches JSON values under additional_metadata. Both
are read when the test's phase report is recorded, so calls made during the test body land
in the same entry.
add_account(account)
Adds an account to the test report manually. Useful when AccountUnderTest use are not automatically detected in function arguments.
add_report_metadata(report_metadata)
Adds any provided json-friendly metadata to a test's report.
Raises:
| Type | Description |
|---|---|
CatsUsageError
|
A value is not JSON-serializable. Evidence is written verbatim or not
at all, so convert values (for example |
Standards registry and the other hooks
A conftest may implement three CATS hooks: pytest_cats_standards (the registry below),
pytest_cats_test_pack (see Test packs) and pytest_cats_provides(cats_config),
which returns the requirements the deployment provides for tests marked @cats_requires
(see Test selection).
CATS pytest hook specifications.
Consumers implement these hooks in their conftest.py. Registration happens via pytest_addhooks in cats/pytest_plugin.py.
StandardMetadata
dataclass
Per-standard metadata: the controls a standard attests to, for every report format that names them.
Attributes:
| Name | Type | Description |
|---|---|---|
control_ids |
list[str]
|
Framework control IDs that this standard attests to
(e.g. |
control_group |
str | None
|
Optional free-form grouping label (e.g. |
CatsHookSpec
Hook specifications for CATS pytest plugin extensions.
pytest_cats_provides(cats_config)
Return the requirements this deployment provides, so a test pack can supply its defaults.
A test decorated @cats_requires(...) runs only when every requirement it names is
provided. test_selection.provides in cats.yaml is the user's half; this hook is the
pack's: return what the deployment satisfies by default, or infer it. Every implementer's
result is unioned with provides before selection, and None contributes nothing.
cats_config is the loaded configuration, so a default can depend on what the user
declared::
from cats import CatsConfig
def pytest_cats_provides(cats_config: CatsConfig) -> set[str]:
if "container-deployment" in cats_config.test_selection.provides:
return set()
return {"standard-deployment"}
Called once per collecting process, during collection and after every conftest under the
collected paths is loaded, so any of them may implement it; under pytest-xdist each worker
collects and calls it. An implementation that raises, or returns anything other than a
set, list or tuple of non-empty strings, is warned about and provides nothing: the tests
that needed what it would have provided are then recorded not_run with the gap named,
which is visible in the terminal summary and as counts.not_run in report.json.
pytest_cats_standards()
Return a registry mapping standard_name to StandardMetadata.
Called once per session on the main process, after collection and
before the first test runs. Values here are used when a @cats(...)
decorator does not supply control_ids or control_group directly.
Every implementer's registry counts: a shared organization conftest, an
installed plugin and a project conftest are merged. Two implementers may
declare the same standard only with the same value; different values are
a usage error naming both, and the session stops before any test runs.
Every control id must be a non-empty string; with oscal_ar in
reporting.report_formats it must also be an OSCAL token (see
:class:StandardMetadata). Either failure is reported the same way.
Implement it in a conftest pytest loads before collection (the root
conftest.py, a test*/ directory's, or a directory passed to
pytest). Under pytest-xdist the main process does not collect, so a
conftest first found during collection is warned about and its registry
is not in the reports.
Return None to opt out entirely — decorator values and the
standard_name fallback still apply.
pytest_cats_test_pack()
Declare the test pack this conftest belongs to.
CATS identifies itself in the User-Agent of its AWS API calls as
AwsCats/<version> (see "Usage attribution" in the README). Return a
:class:~cats.user_agent.CatsTestPack and CATS appends the pack's name and
version the same way, as AwsCatsPack/<name>#<version>, with any metadata
as further #key#value pairs, and records the pack in run.test_packs of
report.json (the same token is the OSCAL cats-test-pack prop and the JUnit
cats:test_pack property)::
from cats import CatsTestPack
def pytest_cats_test_pack() -> CatsTestPack:
return CatsTestPack(name="acme-controls", version="0.99.4")
Read once per process in CATS's pytest_configure, so the implementing conftest
must be one pytest loads before configure: the root conftest.py, a test*/
directory's conftest, or one under a path given to pytest. CATS warns about one
loaded later. Every implementer's result counts; a repeated name keeps its first
declaration. Values must identify a release, never a run. An invalid result is
skipped with a warning.
Test packs
The User-Agent suffix CATS adds to its AWS API calls, and the one place it is stamped.
CATS identifies itself as AwsCats/<version>, as disclosed in the README. A test pack may
add its own name and version through the pytest_cats_test_pack hook (cats.hookspecs);
the pytest plugin registers the result here at configure time and every session built
afterwards carries both tokens. Token content is limited to RFC 7230 tchar characters
because botocore appends user_agent_extra to the header without sanitizing it.
CatsTestPack
pydantic-model
Bases: BaseModel
A test pack's identity, rendered as one User-Agent token.
name and metadata keys are lowercase [a-z0-9._-]; version and metadata values
are [A-Za-z0-9._+-]. # separates segments and may not appear inside one. Values
identify a release, never a run.
The model is frozen and metadata is a read-only mapping, so a registered pack cannot
change after the hook returned it. Constructing one with a field that breaks these rules
raises pydantic's ValidationError (the one CATS export whose errors are not a
CatsError).
Example::
CatsTestPack(name="acme-controls", version="0.99.4")
# -> AwsCatsPack/acme-controls#0.99.4
Show JSON schema:
{
"additionalProperties": false,
"description": "A test pack's identity, rendered as one User-Agent token.\n\n``name`` and metadata keys are lowercase ``[a-z0-9._-]``; ``version`` and metadata values\nare ``[A-Za-z0-9._+-]``. ``#`` separates segments and may not appear inside one. Values\nidentify a release, never a run.\n\nThe model is frozen and ``metadata`` is a read-only mapping, so a registered pack cannot\nchange after the hook returned it. Constructing one with a field that breaks these rules\nraises pydantic's ``ValidationError`` (the one CATS export whose errors are not a\n``CatsError``).\n\nExample::\n\n CatsTestPack(name=\"acme-controls\", version=\"0.99.4\")\n # -> AwsCatsPack/acme-controls#0.99.4",
"properties": {
"name": {
"pattern": "^[a-z0-9._-]{1,64}$",
"title": "Name",
"type": "string"
},
"version": {
"pattern": "^[A-Za-z0-9._+-]{1,64}$",
"title": "Version",
"type": "string"
},
"metadata": {
"patternProperties": {
"^[a-z0-9._-]{1,64}$": {
"pattern": "^[A-Za-z0-9._+-]{1,64}$",
"type": "string"
}
},
"title": "Metadata",
"type": "object"
}
},
"required": [
"name",
"version"
],
"title": "CatsTestPack",
"type": "object"
}
Config:
frozen:Trueextra:forbidstrict:True
Fields:
-
name(Segment) -
version(Value) -
metadata(Mapping[Segment, Value])
Validators:
-
_freeze_metadata→metadata -
_token_fits
Type alias and version
GetAccountFixtureisCallable[[str], AccountUnderTest]: the type of theget_account_fixturefixture, which resolves an account fixture by name and records the account in the test's evidence. See multi-account fixtures.__version__is the installed CATS release as a string, the valuecats --versionprints and thecats:versionstack tag carries.