How CATS Works
The Lifecycle
- Configure: Define accounts and network targets in
cats.yaml - Deploy: CATS creates CloudFormation stacks with IAM roles and optional network infrastructure
- Test: pytest runs, CATS fixtures assume roles and interact with deployed resources
- Destroy: Remove all CATS infrastructure
Phase 1: Configuration
cats.yaml defines three things:
# Where CATS runs from and how it deploys
cats_deployment_settings:
parent_account_id: "999999999999" # Account running tests
aws_region: us-east-1
# What accounts to test
accounts_under_test:
prod: # Config key → fixture name
account_id: "111111111111"
account_network_under_test: # Optional: network targets
private:
vpc_id: vpc-abc123
subnet_ids: [subnet-111]
The config key (prod) becomes your fixture name (prod_account).
Phase 2: Deploy
cats infra deploy prod creates a CloudFormation stack in the target account:
Always created:
- IAM role trusting your parent account
- S3 bucket for UDP test receipts
Created per network target (if account_network_under_test defined):
- EC2 instance listening on configured ports
- One probe Lambda per network target for HTTP/TCP/UDP calls (if the
lambdahandler is enabled)
CloudFormation outputs store metadata (instance IDs, DNS names, bucket names) that CATS reads at test time.
Deploy returns when the stack is complete, the test role is usable, and each EC2 listener is ready,
meaning bound to its ports.
CloudFormation considers an instance complete when it is running, before its user data has started
the listeners, so each listener's user data waits for its ports and writes a ready line to its serial
console, and CATS reads that line through the EC2 API before reporting success. No network path from
the instance is needed, so a listener in a fully isolated subnet is gated like any other. The first
cats test never meets a listener that is still starting.
CloudFormation likewise considers an IAM role and its policy complete before IAM's authorizer has
converged on them, a window of a few seconds in which a call made as the role is denied. So the deploy
assumes the test role from the same credentials cats test will use and makes the first call a test
session makes, cloudformation:DescribeStacks on the stack, retrying a denial until it succeeds. That
proves the trust policy and CATS's own policy on the role are live; each additional role is proven
assumable. The grants you attach yourself are written earlier in the stack through the same IAM path, so
they are expected to be live too, but the deploy does not prove them: there is no call CATS could know
is safe to make under an arbitrary policy. A re-run of cats infra deploy with nothing to change runs
the same checks, so it also means ready.
Phase 3: Test
When you run cats test, the pytest plugin:
- Loads
cats.yamland registers fixtures for each account - When a test requests
prod_account, CATS:- Assumes the IAM role in that account
- Reads CloudFormation stack outputs
- Returns an
AccountUnderTestobject with a boto3 session
def test_something(prod_account: AccountUnderTest) -> None:
# prod_account.session is a boto3 Session with assumed credentials
s3 = prod_account.role.readonly.client("s3")
# For network tests, access targets by name
result = prod_account.private.http_call(target, 443)
Control reporting
The run's output is a report, not a pytest summary. @cats(standard_name=...) names the control
a test proves, and every entry records that control, the accounts the test used, and the outcome,
in every format listed under report_formats. See
Reporting and Compliance Alignment.
Identity Model
CATS uses two distinct identities:
| Identity | Used For | Privilege Level |
|---|---|---|
| Deploy identity | cats infra deploy/destroy |
High (creates IAM roles, EC2, Lambda) |
| Test identity | Test execution | Scoped to test needs |
The deploy identity is typically AWSControlTowerExecution or OrganizationAccountAccessRole, roles that already exist in your accounts. You can also create a narrower role of your own for this purpose.
The test identities are created by CATS with least-privilege permissions for the deployed infrastructure.
Treat infrastructure as ephemeral
CATS infrastructure is designed to be temporary:
This should happen every test cycle. In CI/CD, use finally blocks to ensure cleanup even on failure.
Warning
CATS does not automatically clean up. Orphaned EC2 instances cost money and create security surface area.
Data Flow Diagram
┌─────────────────────────────────────────────────────────────────┐
│ cats.yaml │
│ accounts_under_test: │
│ prod: ← config key │
│ account_id: "111..." │
│ account_network_under_test: │
│ private: ← network target name │
│ vpc_id: vpc-abc │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ CloudFormation Stack (in target account) │
│ Outputs: │
│ Ec2DnsNameprivate: "ip-10-0-1-5.ec2.internal" │
│ ResponseBucketName: "cats-test-infra-prod-catsresponse..." │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ pytest fixture: prod_account │
│ .session → boto3 Session with assumed credentials │
│ .account_id → "111111111111" │
│ .private → NetworkHandler with EC2 DNS, methods │
│ .http_call() → CatsNetworkResult │
│ .tcp_call() → CatsNetworkResult │
│ .udp_call() → CatsNetworkResult (with correlation_id)│
│ .receive_s3(sent) → bool │
│ .shell() (requires account_settings.allow_shell: true at deploy time) │
│ .ec2_dns_name │
└─────────────────────────────────────────────────────────────────┘
Network Testing
When you define account_network_under_test, CATS deploys infrastructure to make and receive network calls:
- EC2 listener: Receives HTTP/TCP/UDP traffic, stores UDP receipts in S3
- Probe Lambda (optional): Makes outbound calls from within the VPC
Both handlers run the same probe source, so they classify outcomes identically.
Every call returns a CatsNetworkResult: a CatsNetworkResponse category
(SUCCESS, REFUSED, TIMEOUT, DNS_FAILURE, TLS_FAILURE, FAILURE, SENT)
plus the reason, http_status, and correlation_id evidence behind it.
See Network Testing for full details on handlers, protocols, and VPC endpoints.
The same behavior is also reachable without pytest — as a stateless CLI (cats run) and an MCP server (cats mcp) for AI agents. See Agentic.
This lets you prove traffic flows or is blocked:
def test_traffic_flows(prod_account: AccountUnderTest, dev_account: AccountUnderTest) -> None:
"""Prove prod can reach dev on port 443."""
target = dev_account.private.ec2_dns_name
result = prod_account.private.http_call(target, 443)
assert result == CatsNetworkResponse.SUCCESS
def test_traffic_blocked(prod_account: AccountUnderTest) -> None:
"""Prove prod cannot reach the internet."""
result = prod_account.private.http_call("example.com", 443)
assert result == CatsNetworkResponse.TIMEOUT