Skip to content

Helpers

Every name below is exported from cats; import it as from cats import .... The module paths shown by the renderer are internal and may move.

Results and helpers

Helper functions and result types for use in tests, alongside the fixtures.

get_account_fixture_names and get_accounts_by_tag drive test parameterization across accounts.

CatsNetworkResponse

Bases: StrEnum

Outcome of a network call, classified by what happened on the wire.

Each value is a precise claim about what the probing endpoint observed. Which control produced it is interpretation, and belongs in the test's docstring.

  • SUCCESS: Connection established. For HTTP/HTTPS, any HTTP response (including 4xx/5xx and 3xx, which are not followed) is SUCCESS - the call arrived. The status code is on CatsNetworkResult.http_status for tests that care.
  • REFUSED: RST received (ECONNREFUSED or ECONNRESET) - the network path is OPEN but nothing accepted the connection. For HTTP, a listener that accepted the connection and closed it without any response is also REFUSED. On a blocked-traffic test this means the control is NOT working. Indistinguishable from a Network Firewall REJECT.
  • TIMEOUT: The TCP connect got no answer within the budget - consistent with a silent drop. This is the expected outcome for SG/NACL/Network Firewall DROP/ route-table isolation tests, but the endpoint cannot distinguish policy enforcement from transient loss: all silent drops look the same.
  • DNS_FAILURE: Name resolution failed - the resolver answered NXDOMAIN/NODATA or was unreachable. What Route 53 Resolver DNS Firewall blocks produce, and also what a typo produces. No packet was ever sent to the target.
  • TLS_FAILURE: TCP connected, the TLS handshake failed - untrusted or mismatched certificate, a plaintext listener on the port, or a protocol mismatch. What TLS inspection with an untrusted CA produces. The OpenSSL text is in reason.
  • FAILURE: The path could not be proven open or blocked. Either a connect that succeeded with no HTTP reply inside the budget (a hung backend, NOT a silent drop; reason starts "connected; no HTTP reply"), or an unclassified network error (ENETUNREACH, EHOSTUNREACH, EADDRNOTAVAIL, ...; the errno is in reason). Never an expected value.
  • SENT: UDP datagram left the host (connectionless - use receive_s3 to confirm arrival). REFUSED and TIMEOUT are unreachable for UDP.

CatsNetworkResult dataclass

Outcome of a network call: a CatsNetworkResponse plus the evidence behind it.

Equality and hashing are category-only, against another result or a bare CatsNetworkResponse: result == CatsNetworkResponse.TIMEOUT is the assertion to write. reason, http_status, and correlation_id are diagnostics: they are excluded from equality so error strings never become load-bearing, and they are printed by the repr so a failing assertion is self-diagnosing.

Comparing to a plain string is refused (result == "TIMEOUT" is False): the enum is the contract.

Attributes:

Name Type Description
response CatsNetworkResponse

The classified outcome.

reason str | None

str(exception) from the probe for non-SUCCESS/SENT outcomes.

http_status int | None

The HTTP status code, only for HTTP/HTTPS SUCCESS.

correlation_id str | None

The UDP receipt ID that was sent or attempted, only for UDP.

value property

The response category as a string, for CLI/JSON consumers.

ShellResult pydantic-model

Bases: BaseModel

Result of NetworkHandler.shell().

stdout and stderr are stripped of trailing whitespace. stdout_truncated and stderr_truncated are set when SSM returned exactly its maximum for that stream (24,000 characters of stdout, 8,000 of stderr, before stripping): SSM keeps the first N characters and gives no other signal, so an output of exactly N characters reads as truncated and the remainder is not recoverable through CATS. Write large output to S3 from the command instead.

Show JSON schema:
{
  "description": "Result of NetworkHandler.shell().\n\n``stdout`` and ``stderr`` are stripped of trailing whitespace. ``stdout_truncated`` and\n``stderr_truncated`` are set when SSM returned exactly its maximum for that stream (24,000\ncharacters of stdout, 8,000 of stderr, before stripping): SSM keeps the first N characters\nand gives no other signal, so an output of exactly N characters reads as truncated and the\nremainder is not recoverable through CATS. Write large output to S3 from the command instead.",
  "properties": {
    "stdout": {
      "title": "Stdout",
      "type": "string"
    },
    "stderr": {
      "title": "Stderr",
      "type": "string"
    },
    "exit_code": {
      "title": "Exit Code",
      "type": "integer"
    },
    "duration_ms": {
      "title": "Duration Ms",
      "type": "integer"
    },
    "stdout_truncated": {
      "title": "Stdout Truncated",
      "type": "boolean"
    },
    "stderr_truncated": {
      "title": "Stderr Truncated",
      "type": "boolean"
    }
  },
  "required": [
    "stdout",
    "stderr",
    "exit_code",
    "duration_ms",
    "stdout_truncated",
    "stderr_truncated"
  ],
  "title": "ShellResult",
  "type": "object"
}

Fields:

  • stdout (str)
  • stderr (str)
  • exit_code (int)
  • duration_ms (int)
  • stdout_truncated (bool)
  • stderr_truncated (bool)

get_account_fixture_names()

Get fixture names for all configured accounts.

get_accounts_by_tag(key, value=None)

Return fixture names for accounts with this tag.

Parameters:

Name Type Description Default
key str

Tag key to match

required
value str | None

If provided, tag must have this value. If None, tag just needs to exist.

None

Returns:

Type Description
list[str]

List of fixture names (e.g., ["prod_account", "dev_account"])

Reporting

CatsReporter

The evidence a test adds to its own report entry.

One instance per test, provided by the function-scoped cats_report fixture (and by get_account_fixture, which records the accounts it resolves). add_account puts an account into accounts_in_test when the test reached it some way other than a fixture argument; add_report_metadata attaches JSON values under additional_metadata. Both are read when the test's phase report is recorded, so calls made during the test body land in the same entry.

add_account(account)

Adds an account to the test report manually. Useful when AccountUnderTest use are not automatically detected in function arguments.

add_report_metadata(report_metadata)

Adds any provided json-friendly metadata to a test's report.

Raises:

Type Description
CatsUsageError

A value is not JSON-serializable. Evidence is written verbatim or not at all, so convert values (for example value.isoformat() for datetimes) before adding them.

Standards registry and the other hooks

A conftest may implement three CATS hooks: pytest_cats_standards (the registry below), pytest_cats_test_pack (see Test packs) and pytest_cats_provides(cats_config), which returns the requirements the deployment provides for tests marked @cats_requires (see Test selection).

CATS pytest hook specifications.

Consumers implement these hooks in their conftest.py. Registration happens via pytest_addhooks in cats/pytest_plugin.py.

StandardMetadata dataclass

Per-standard metadata: the controls a standard attests to, for every report format that names them.

Attributes:

Name Type Description
control_ids list[str]

Framework control IDs that this standard attests to (e.g. ["AC-3", "SC-7", "1.1.1"]), carried as written into report.json, JUnit and Security Hub RelatedRequirements. With oscal_ar in reporting.report_formats each ID becomes the target-id on an OSCAL finding, which the schema constrains to a token: letters, digits, ., -, _, starting with a letter or _; write SC-7(5) in the catalog convention sc-7.5 there. The list is space-joined into observation.remarks so OSCAL consumers that scan free-text for catalog IDs can extract them.

control_group str | None

Optional free-form grouping label (e.g. "Governance") emitted as observation.props[name="control-group"]. OSCAL consumers can use this to group observations by domain.

CatsHookSpec

Hook specifications for CATS pytest plugin extensions.

pytest_cats_provides(cats_config)

Return the requirements this deployment provides, so a test pack can supply its defaults.

A test decorated @cats_requires(...) runs only when every requirement it names is provided. test_selection.provides in cats.yaml is the user's half; this hook is the pack's: return what the deployment satisfies by default, or infer it. Every implementer's result is unioned with provides before selection, and None contributes nothing. cats_config is the loaded configuration, so a default can depend on what the user declared::

from cats import CatsConfig


def pytest_cats_provides(cats_config: CatsConfig) -> set[str]:
    if "container-deployment" in cats_config.test_selection.provides:
        return set()
    return {"standard-deployment"}

Called once per collecting process, during collection and after every conftest under the collected paths is loaded, so any of them may implement it; under pytest-xdist each worker collects and calls it. An implementation that raises, or returns anything other than a set, list or tuple of non-empty strings, is warned about and provides nothing: the tests that needed what it would have provided are then recorded not_run with the gap named, which is visible in the terminal summary and as counts.not_run in report.json.

pytest_cats_standards()

Return a registry mapping standard_name to StandardMetadata.

Called once per session on the main process, after collection and before the first test runs. Values here are used when a @cats(...) decorator does not supply control_ids or control_group directly.

Every implementer's registry counts: a shared organization conftest, an installed plugin and a project conftest are merged. Two implementers may declare the same standard only with the same value; different values are a usage error naming both, and the session stops before any test runs. Every control id must be a non-empty string; with oscal_ar in reporting.report_formats it must also be an OSCAL token (see :class:StandardMetadata). Either failure is reported the same way.

Implement it in a conftest pytest loads before collection (the root conftest.py, a test*/ directory's, or a directory passed to pytest). Under pytest-xdist the main process does not collect, so a conftest first found during collection is warned about and its registry is not in the reports.

Return None to opt out entirely — decorator values and the standard_name fallback still apply.

pytest_cats_test_pack()

Declare the test pack this conftest belongs to.

CATS identifies itself in the User-Agent of its AWS API calls as AwsCats/<version> (see "Usage attribution" in the README). Return a :class:~cats.user_agent.CatsTestPack and CATS appends the pack's name and version the same way, as AwsCatsPack/<name>#<version>, with any metadata as further #key#value pairs, and records the pack in run.test_packs of report.json (the same token is the OSCAL cats-test-pack prop and the JUnit cats:test_pack property)::

from cats import CatsTestPack


def pytest_cats_test_pack() -> CatsTestPack:
    return CatsTestPack(name="acme-controls", version="0.99.4")

Read once per process in CATS's pytest_configure, so the implementing conftest must be one pytest loads before configure: the root conftest.py, a test*/ directory's conftest, or one under a path given to pytest. CATS warns about one loaded later. Every implementer's result counts; a repeated name keeps its first declaration. Values must identify a release, never a run. An invalid result is skipped with a warning.

Test packs

The User-Agent suffix CATS adds to its AWS API calls, and the one place it is stamped.

CATS identifies itself as AwsCats/<version>, as disclosed in the README. A test pack may add its own name and version through the pytest_cats_test_pack hook (cats.hookspecs); the pytest plugin registers the result here at configure time and every session built afterwards carries both tokens. Token content is limited to RFC 7230 tchar characters because botocore appends user_agent_extra to the header without sanitizing it.

CatsTestPack pydantic-model

Bases: BaseModel

A test pack's identity, rendered as one User-Agent token.

name and metadata keys are lowercase [a-z0-9._-]; version and metadata values are [A-Za-z0-9._+-]. # separates segments and may not appear inside one. Values identify a release, never a run.

The model is frozen and metadata is a read-only mapping, so a registered pack cannot change after the hook returned it. Constructing one with a field that breaks these rules raises pydantic's ValidationError (the one CATS export whose errors are not a CatsError).

Example::

CatsTestPack(name="acme-controls", version="0.99.4")
# -> AwsCatsPack/acme-controls#0.99.4
Show JSON schema:
{
  "additionalProperties": false,
  "description": "A test pack's identity, rendered as one User-Agent token.\n\n``name`` and metadata keys are lowercase ``[a-z0-9._-]``; ``version`` and metadata values\nare ``[A-Za-z0-9._+-]``. ``#`` separates segments and may not appear inside one. Values\nidentify a release, never a run.\n\nThe model is frozen and ``metadata`` is a read-only mapping, so a registered pack cannot\nchange after the hook returned it. Constructing one with a field that breaks these rules\nraises pydantic's ``ValidationError`` (the one CATS export whose errors are not a\n``CatsError``).\n\nExample::\n\n    CatsTestPack(name=\"acme-controls\", version=\"0.99.4\")\n    # -> AwsCatsPack/acme-controls#0.99.4",
  "properties": {
    "name": {
      "pattern": "^[a-z0-9._-]{1,64}$",
      "title": "Name",
      "type": "string"
    },
    "version": {
      "pattern": "^[A-Za-z0-9._+-]{1,64}$",
      "title": "Version",
      "type": "string"
    },
    "metadata": {
      "patternProperties": {
        "^[a-z0-9._-]{1,64}$": {
          "pattern": "^[A-Za-z0-9._+-]{1,64}$",
          "type": "string"
        }
      },
      "title": "Metadata",
      "type": "object"
    }
  },
  "required": [
    "name",
    "version"
  ],
  "title": "CatsTestPack",
  "type": "object"
}

Config:

  • frozen: True
  • extra: forbid
  • strict: True

Fields:

  • name (Segment)
  • version (Value)
  • metadata (Mapping[Segment, Value])

Validators:

  • _freeze_metadata → metadata
  • _token_fits

Type alias and version

  • GetAccountFixture is Callable[[str], AccountUnderTest]: the type of the get_account_fixture fixture, which resolves an account fixture by name and records the account in the test's evidence. See multi-account fixtures.
  • __version__ is the installed CATS release as a string, the value cats --version prints and the cats:version stack tag carries.