Security Considerations
CATS requires high privilege to operate across multiple AWS accounts.
Trust Assumptions
CATS assumes:
- Users are authorized to execute tests in target accounts
- Compute environment is controlled (your laptop, your pipeline, your container)
- Pipeline security is your responsibility (CI/CD access controls, secret management)
Privilege Model
Deploy Identity (High Privilege)
Used by cats infra deploy/destroy. Can:
- Create/delete IAM roles and policies
- Create/delete EC2 instances
- Create/delete Lambda functions
- Create/delete S3 buckets
- Modify security groups
Mitigation: Only run cats infra from trusted environments. Use short-lived credentials.
Test Identity (Scoped Privilege)
Created by CATS in each target account. Can:
- Invoke CATS-deployed Lambda functions
- Run SSM commands on CATS-deployed EC2 instances
- Read the CATS response bucket (
s3:GetObject,s3:ListBucket; only the EC2 instance role writes to it) - Read CloudFormation outputs
- Read the account's SSM managed-node inventory (
ssm:DescribeInstanceInformation) - Import findings into the account's own Security Hub (
securityhub:BatchImportFindings), only when thesecurityhubreport format is configured with reporting enabled, and only on that account's default product ARN in the one configured Region: the identity that ran the tests writes their results where the account's owner already looks, and cannot write findings for any other account or product. A finding carries the test's node id, docstring, assertion message, metadata, marker values and account tags, and is readable by everyone and everything with access to that Security Hub; see what a finding contains and who can read it
Mitigation: Test role permissions are scoped to CATS resources by default. The two
exceptions are read-only SSM actions with no resource-level scoping: GetCommandInvocation
reads results of commands the role itself sent, and DescribeInstanceInformation is consulted
only after a SendCommand refusal, to tell a listener that has not registered yet from one
that is registered but refused the command. All of these grants are one inline policy
document on the role, CatsTestRolePolicy, one statement each; cats infra deploy reports
success only after assuming the role and exercising that document, so the role a test run
receives is one IAM has already agreed to.
What a Security Hub finding carries
Each imported finding carries: the test's node id (GeneratorId, cats/nodeid); the test name as
Title; the docstring as Description; the failure text as cats/error_message, truncated to 2048
characters; the skip or exclusion reason as cats/reason; every custom marker value
(cats/marker/*); every add_report_metadata value (cats/metadata/*); the keynames of every
account the test used (cats/accounts_in_test); the account's cats.yaml tags as
Resources[].Tags; and cats/run_id, cats/version and cats/aws_region. In a cross-account
test the same finding, error text included, is imported into every account named in the test. The
full field list is in the Reports reference.
Threat Model
CATS never prints, returns or stores a credential
Every identity CATS uses is a short-lived assume-role session that lives in process memory and
refreshes itself; CATS writes no credential to disk and returns none in any CLI output or MCP tool
result. cats run write-profile and the cats_write_profile MCP tool write an AWS profile that
names a role (role_arn) and the identity to assume it from (source_profile or
credential_source), not keys: the aws CLI or an SDK assumes the role itself. The MCP tool
writes only ~/.aws/config, since an agent acting on untrusted tool output must not be able to
aim the writer at another file; both entry points refuse a file that is not an AWS config file
(one whose section headers are not default, profile ..., sso-session ..., services ...,
plugins or preview) untouched. When a tool
assumes a role through such a profile, the AWS CLI caches the resulting temporary credentials under
~/.aws/cli/cache; that is the CLI's behavior for any role_arn profile and is outside CATS's
control.
Orphaned Infrastructure
Threat: Compute fails mid-run, leaving EC2 instances running.
Impact: Cost (instances keep running), security surface (roles remain assumable).
Mitigation:
- Always use
finally/after_scriptin CI/CD - Run
cats infra statusperiodically to check what is deployed - Set up billing alerts
Misconfigured Tests
Threat: Tests make incorrect assertions, giving false confidence.
Impact: Controls appear to work when they don't.
Mitigation:
- Review test logic carefully
- Test both positive and negative cases
- Validate tests catch known-bad configurations
Dependency Vulnerabilities
Threat: Malicious code in CATS dependencies.
Impact: Arbitrary code execution with your AWS credentials.
Mitigation:
- Every pull request's dependency changes are checked against the GitHub Advisory Database (dependency review) before merge
- The locked runtime closure is audited against known vulnerabilities weekly and on every push to
main(pip-auditin CI), so a disclosure against a version already pinned is caught between releases - Every version is pinned in
uv.lock, and CI installs from the lock, so a build cannot pick up an unreviewed release - Review dependency updates
Compute Environment Compromise
Threat: Attacker gains access to where CATS runs.
Impact: Can assume deploy role, modify infrastructure, exfiltrate data.
Mitigation:
- Secure CI/CD pipelines
- Use short-lived credentials
- Audit CloudTrail for unexpected role assumptions
Malicious Test Injection
Threat: Attacker modifies test code to perform unauthorized actions.
Impact: Tests could create backdoors, exfiltrate data, or destroy resources.
Mitigation:
- Code review for test changes
- Restrict who can modify test code
- Use branch protection
Deployed Infrastructure Security
IAM Roles
- Test roles trust only the configured parent account (
parent_account_id) - Permissions scoped to CATS resources: the test role's actions name the stack's own Lambda, bucket, SSM document and listeners
- No wildcard IAM actions on the roles CATS creates. Two read-only SSM actions have
Resource: "*"because the API offers no resource scoping (ssm:GetCommandInvocation,ssm:DescribeInstanceInformation); the Lambda execution role carries the AWS-managedAWSLambdaVPCAccessExecutionRole; security-group egress is open on both address families so the listener can reach S3 and SSM
Lambda Functions
- Run in your VPC
- Limited to basic network calls
- No internet access unless VPC allows it
EC2 Instances
- Run the current Amazon Linux AMI CATS selects at deploy time (or your
existing_ec2_ami_id) with SSM agent - Listen on configured ports only, as an unprivileged transient user with a read-only filesystem
- Security group ingress scoped to the configured listen ports: one IPv4 and one IPv6 rule per port, from any source, nothing else inbound
- No public IPv4 address, whatever the subnet's
MapPublicIpOnLaunchsetting: the interface is declared withAssociatePublicIpAddress: false, so over IPv4 the listen ports are reachable only from inside the VPC and whatever routes into it, and an internet gateway carries nothing for the listener. It reaches SSM through NAT, a route to the organization's endpoints, or endpoints in its VPC (choosing the handler). In a subnet that assigns IPv6 addresses the listener's address is globally routable, and a::/0route to an internet gateway exposes the listen ports to the internet; attachexisting_security_group_idsthere (security groups on listeners) - UDP listeners require a per-account receipt token before writing to S3; TCP/HTTP listeners answer with a fixed, content-free reply (today an empty
200 OK) and disclose no content
S3 Buckets
- Require HTTPS (TLS)
- Block public access
- Used only for UDP receipt storage
VPC Endpoints
- Created only where
cats.yamlnames them (gateway_endpoints,interface_endpoints), in the VPC under test - Every endpoint carries a policy allowing only principals in the account under test (
aws:PrincipalAccount), where every CATS caller of the endpoint lives; an endpoint with no policy would take AWS's default, which admits any principal anywhere. The account's other workloads are unaffected, since an interface endpoint with private DNS carries the whole VPC's traffic for its service - A shared VPC consumed from other accounts needs an organization condition instead, which CATS does not offer; see the endpoint policy
- Interface endpoints are on the group
interface_endpoint_security_group_idsnames, or the VPC's default group when it is unset; CATS never creates a group for them. A stock default group admits only its own members, so the setting is what lets the listener reach a CATS-created endpoint (security groups on interface endpoints)
UDP receipt token
The EC2 listeners accept traffic on all configured ports — an application-layer control cannot close a port (the kernel completes the handshake or receives the datagram before any CATS code runs). What each layer controls:
| Concern | Control |
|---|---|
| Port is reachable at all | Security groups: CATS-managed listener groups admit only the configured listen ports, one IPv4 and one IPv6 rule per port |
| Unauthenticated side effects (UDP → S3 write) | UDP receipt token: a valid {token}: message prefix is required before any S3 write. Always enforced, no opt-out |
| Unauthenticated data disclosure (TCP/HTTP banner) | Banner removed: any received data is answered with a fixed, content-free reply (today an empty 200 OK) |
The token is generated per account at create_stack, carried forward unchanged
on updates, and exposed as the UdpReceiptToken stack output. It is a
network-abuse guard, not a bearer credential for anything IAM-protected —
its threat model is unauthenticated network actors, not principals with AWS
read access. Any principal with cloudformation:DescribeStacks on the stack,
including the reduced-privilege CATS test role, can read the token — this is
intended, so secret scanners flagging the plaintext CFN output are reporting
expected behavior. It is equally visible to anyone who can read the listener's
user data (ec2:DescribeInstanceAttribute) or the stack's parameters, and to
any process on the listener itself, where it is an argument on the udpserver
command line. CATS callers never handle it: udp_call prepends it
automatically and never transmits it to non-CATS targets.
Rotation is destroy + deploy (a fresh token is generated on create), matching the ephemeral lifecycle.
Shell access (allow_shell)
Off by default at the account level. When account_settings.allow_shell: true at deploy time, CATS extends the test role's SSM policy with the run-command document shell() uses (today AWS's AWS-RunShellScript), which lets NetworkHandler.shell() run arbitrary commands on the EC2 listeners of that account's CATS stack.
What this protects against. With the flag off at deploy time, a compromised test role cannot run arbitrary commands on CATS EC2 instances: only the probe document is invocable. This is a real IAM reduction for the test role's standing permissions.
What this does NOT protect against.
- A compromised deploy identity. Anyone able to edit
cats.yamland runcats infra deploycan flipallow_shelltotrueand redeploy in seconds. The deploy identity is the real boundary; this flag sits well inside it. - Test authors (human or LLM) who can edit
cats.yaml. The Python guard onNetworkHandler.shell()is a convenience — it reads the livecats.yamland surfaces a clear error. It is not a security control.
Blast radius when enabled.
- Commands run as the SSM agent's command user, which on the Amazon Linux images CATS deploys today is root.
- They inherit the EC2 instance's IAM role, not the CATS test role. The role CATS creates grants exactly what SSM Agent needs to register and run commands (
ssm:UpdateInstanceInformation, thessmmessagesandec2messageschannel actions) pluss3:PutObjecton the response bucket, so a shell on the listener gains no read of SSM parameters, no write to SSM inventory or compliance data, and no access outside the bucket. Supplyexisting_ec2_instance_profile_nameto use a profile of your own instead; anything that profile grants, a shell inherits. - Instance targeting is scoped to the instances of the same stack (the
cats:stacktag), so two CATS stacks in one account cannot reach each other's listeners. - No command allow-list. Enable only where callers and deploy identity are trusted.
The flag is equivalent to granting the test role ssm:SendCommand on the run-command document yourself; it removes that boilerplate.
Reporting Security Issues
If you discover a potential security issue in CATS, notify AWS/Amazon Security via the vulnerability reporting page or email aws-security@amazon.com — see SECURITY.md. Please do not create a public GitHub issue.