s2n-quic Continuous Integration

s2n-quic executes a comprehensive suite of tests, linters, benchmarks, simulations and other checks on each pull request and merge into main. Information about these checks is provided below.

Tests

s2n-quic defines many tests that can be executed with cargo test. These tests, described below, are executed across a variety of operating systems, architectures, and Rust versions.

Unit Tests

Unit tests validate the expected behavior of individual components of s2n-quic. Typically, unit tests will be located in a tests module at the end of the file being tested. When there are a significant number of unit test functions for given component, the tests module may be located in a separate file.

Integration Tests

s2n-quic integration tests use the public API to validate the end-to-end behavior of the library under specific scenarios and configurations. Integration tests are located in the top-level s2n-quic crate, in the tests module.

Snapshot Tests

Snapshot tests use insta to assert complex output remains consistent with expected references values. In s2n-quic, snapshot tests are typically constructed in one of two ways:

  • using the insta::assert_debug_snapshot macro to compare the Debug representation of an instance to the snapshot
  • using the event::testing::Publisher::snapshot() event publisher to assert against a snapshot of events emitted by the s2n-quic event framework

Property Tests

Property tests assert that specific properties of the output of functions and components are upheld under a variety of inputs. s2n-quic uses the Bolero property-testing framework to execute property-testing with multiple fuzzing engines as well as the Kani Rust Verifier. For more details on how Bolero and Kani are used together in s2n-quic, see the following blog posts from the Kani Rust Verifier Blog:

Fuzz Tests

Fuzz tests provide large amounts of varied input data to assert s2n-quic behaves as expected regardless of the input. Fuzz testing in s2n-quic comes in three flavors:

  • Component-level fuzz testing using the Bolero property-testing framework. These tests generate a corpus of inputs that is included in the s2n-quic repository to allow the fuzz tests to be replayed when executing cargo test.
  • End-to-end QUIC protocol-level fuzzing using quic-attack. quic-attack is a collection of features that collectively turn s2n-quic into an online QUIC protocol fuzzer. It allows incoming and outgoing datagrams and packets, as well as the port number on incoming datagrams, to be intercepted and manipulated.
  • UDP protocol-level fuzzing using udp-attack. udp-attack generates random UDP packets and transmits them to s2n-quic to catch issues with packet handling.

Concurrency Permutation Tests

loom is used to validate the behavior of concurrent code in s2n-quic. loom executes concurrent code using a simulation of the operating system scheduler and Rust memory model to evaluate concurrent code under all possible thread interleavings.

Interoperability

The quic-interop-runner defines a suite of test cases that ensure compatibility between QUIC implementations. s2n-quic publishes a report with the results.

Compliance

s2n-quic annotates source code with inline references to requirements in IETF RFC specifications. Duvet is used to generate a report, which makes it easy to track compliance with each requirement.

Simulations

A Monte Carlo simulation tool is used to execute thousands of randomized simulations of s2n-quic that vary one or more network variables, such as bandwidth, jitter, and round trip time. The report output provides a visual representation of the relationship between the input variables and overall performance.

A loss recovery simulation tool plots the growth of the congestion window over time under various simulated loss scenarios and publishes the results in a report to visualize changes to the congestion control algorithm.

Performance & Efficiency Profiling

Flame graphs are generated and published in a report to visualize stack traces produced under a variety of data transfer scenarios.

dhat performs heap profiling and publishes the results in a report.

Clippy

clippy is a rust linter which catches common mistakes.

Rustfmt

rustfmt ensures code is consistently formatted.

Miri

Miri detects undefined behavior and memory leaks.

Code Coverage

LLVM source-based code coverage measures how much of s2n-quic code is executed by tests. s2n-quic publishes a report with the results.