Automated Tests

Guide to this SDK's automated testing setup.

The SDK's GitHub repo relies on the following tools to provide automated testing of commits and pull requests.

• Travis CI is the primary service used for testing.
  See: aws-iot-device-sdk-embedded-C on Travis CI
• Codecov provides code coverage results.
  See: aws-iot-device-sdk-embedded-C on Codecov

The following files on the GitHub repo control the automated tests:

• .travis.yml
  Provides the job matrix to run on Travis CI.
• scripts/ directory
  Shell scripts (sh) that run on Travis CI servers.
  • The scripts that test the libraries are named after the library they test. For example, ci_test_mqtt.sh tests the MQTT library.
  • ci_test_doc.sh tests the documentation build. Runs for pull requests only.
  • ci_test_coverage.sh generates and submits coverage results to Codecov. Runs for commits only.

Setup

How to set up the automated testing services.

All of the automated testing services have web interfaces that can be connected to a GitHub account.

Travis CI

How to set up Travis CI.

Link to Travis CI: https://travis-ci.org/

1. Click Sign in with GitHub in the upper right corner to log in with your GitHub account. On your first login, Travis CI will prompt for GitHub permissions access.
2. Once signed in, hover over your GitHub account in the upper right corner. Click Settings in the menu that pops up.
3. On the Settings page, toggle the switch next to your fork of the C SDK to On.
4. Travis CI is now set up. See Setting the credentials on Travis CI for adding commit credentials to Travis CI.

Codecov

How to set up Codecov.

Link to Codecov: https://codecov.io/

1. Click Log In in the upper right corner. Then select Log in with GitHub to log in with your GitHub account. On your first login, Codecov will prompt for GitHub permissions access.
2. Navigate to codecov.io/gh/your GitHub username/aws-iot-device-sdk-embedded-C
3. Click on Settings, then copy the Repository Upload Token displayed.
4. Set the Travis CI environment variable CODECOV_TOKEN to the Repository Upload Token. Travis CI will automatically submit coverage data on commit check builds.

Commit vs. Pull Request

The automated tests distinguish between GitHub commits and pull requests. Different sets of tests run on commits and pull requests.

Because pull requests come from other repos and contain unknown code, Travis CI will not provide encrypted environment variables for pull requests. Therefore, pull requests are tested against unsecured servers. The following tests are run for pull requests.

• Common (ci_test_common.sh): this script tests common components like the task pool, atomics, etc. These tests do not use the network.
• MQTT (ci_test_mqtt.sh): the MQTT tests run on an unsecured connection against a locally-installed Mosquitto broker on the Travis CI VM.
• Shadow (ci_test_shadow.sh): only the Shadow unit tests are run for a pull request build. These tests do not use the network.
• Documentation (ci_test_doc.sh): checks that the documentation builds against the code in the pull request.

For commits, the credentials necessary for connecting to AWS IoT are provided by Travis CI. The following tests are run for commits.

• Common (ci_test_common.sh): runs the same tests as the common pull request tests.
• MQTT (ci_test_mqtt.sh): runs the same MQTT tests as the MQTT pull request tests, but against the AWS IoT MQTT broker instead of a Mosquitto broker.
• Shadow (ci_test_shadow.sh): runs both the Shadow unit tests and the Shadow system tests. Requires connection to AWS IoT.
• Coverage (ci_test_coverage.sh): Gathers code coverage data and submits it to Codecov.

Note

Rarely, the Shadow tests on a GitHub fork will fail with a 429 TOO MANY REQUESTS error. If this happens, restart the Shadow tests. The main C SDK GitHub repo has an increased limit for Shadow requests, so it should not encounter this error.

Because different sets of tests are run for commits and pull requests, both commit and pull request checks should be used to validate code changes. The commit checks can be set up to run on a GitHub fork, where they will run whenever a new commit is pushed to the fork. When a pull request is made from the fork to the main repo, the pull request checks will run.

AWS Credentials and Policy

How to set up AWS IoT credentials for commit checks.

Forking the main GitHub repo copies the Travis CI scripts, but does not copy any credentials, which are provided by Travis CI's encrypted environment variable feature. AWS IoT credentials are required to run the commit checks. No credentials are required to run pull request checks.

Formatting the client certificate and private key

The AWS IoT client certificate and private key must be formatted to be acceptable for use in a CI script.

The credentials are written from Travis encrypted environment variables using the following command:

echo -e $AWS_IOT_CLIENT_CERT > credentials/clientCert.pem

Therefore, all given credentials must be formatted for echo -e. As an example, consider the following certificate:

-----BEGIN CERTIFICATE-----
MIIB8TCCAZugAwIBAwIBADANBgkqhkiG9w0BAQQFADBfMRMwEQYDVQQDEwpuZXRh
cHAuY29tMQswCQYDVQQGEwJVUzEJMAcGA1UECBMAMQkwBwYDVQQHEwAxCTAHBgNV
BAoTADEJMAcGA1UECxMAMQ8wDQYJKoZIhvcNAQkBFgAwHhcNMTAwNDI2MTk0OTI4
WhcNMTAwNTI2MTk0OTI4WjBfMRMwEQYDVQQDEwpuZXRhcHAuY29tMQswCQYDVQQG
EwJVUzEJMAcGA1UECBMAMQkwBwYDVQQHEwAxCTAHBgNVBAoTADEJMAcGA1UECxMA
MQ8wDQYJKoZIhvcNAQkBFgAwXDANBgkqhkiG9w0BAQEFAANLADBIAkEAyXrK2sry
-----END CERTIFICATE-----

It should be formatted as follows for Travis CI:

-----BEGIN\ CERTIFICATE-----\\nMIIB8TCCAZugAwIBAwIBADANBgkqhkiG9w0BAQQFADBfMRMwEQYDVQQDEwpuZXRh\\ncHAuY29tMQswCQYDVQQGEwJVUzEJMAcGA1UECBMAMQkwBwYDVQQHEwAxCTAHBgNV\\nBAoTADEJMAcGA1UECxMAMQ8wDQYJKoZIhvcNAQkBFgAwHhcNMTAwNDI2MTk0OTI4\\nWhcNMTAwNTI2MTk0OTI4WjBfMRMwEQYDVQQDEwpuZXRhcHAuY29tMQswCQYDVQQG\\nEwJVUzEJMAcGA1UECBMAMQkwBwYDVQQHEwAxCTAHBgNVBAoTADEJMAcGA1UECxMA\\nMQ8wDQYJKoZIhvcNAQkBFgAwXDANBgkqhkiG9w0BAQEFAANLADBIAkEAyXrK2sry\\n-----END\ CERTIFICATE-----

Note the spaces have an escape character \ before them, as in -----BEGIN\ CERTIFICATE-----. Line breaks have been removed and replaced with \\n.

Similar formatting should be applied to the matching private key.

Setting the credentials on Travis CI

The AWS IoT endpoint, client certificate, and client certificate private key are set on Travis CI's "Settings" page.

To edit the credentials, go to the "Settings" page on Travis CI. This is usually <repo URL>/settings; for example, https://travis-ci.org/aws/aws-iot-device-sdk-embedded-C/settings for the main repo. Under "Environment Variables", set the following variables:

Name	Value	Notes
AWS_IOT_CLIENT_CERT	Formatted client certificate	The AWS IoT client certificate. This value does not need to be hidden in the build log.
AWS_IOT_ENDPOINT	ABCDEFG1234567.iot.<region>.amazonaws.com	The AWS IoT endpoint. This value does not need to be hidden in the build log.
AWS_IOT_PRIVATE_KEY	Formatted private key	The private key matching the AWS IoT client certificate. Attention This value should be hidden in the build log!
IOT_IDENTIFIER_PREFIX	CSDKCI	This string is prepended to all MQTT client identifiers used by the CI. It may be anything, but must match the client identifier in the AWS IoT policy. The example policy below uses CSDKCI. This value does not need to be hidden in the build log.

Setting the AWS IoT policy

The certificates used with Travis CI must have an appropriate policy attached.

AWS recommends using the most restrictive policy possible. The following policy should be used for the CI.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": [
        "arn:aws:iot:<region>:<account>:client/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive"
      ],
      "Resource": [
        "arn:aws:iot:<region>:<account>:topic/${iot:ClientId}/*",
        "arn:aws:iot:<region>:<account>:topic/*/LastWillAndTestament",
        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/shadow/*",
        "arn:aws:iot:<region>:<account>:topic/$aws/things/${iot:ClientId}/jobs/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": [
        "arn:aws:iot:<region>:<account>:topicfilter/${iot:ClientId}/*",
        "arn:aws:iot:<region>:<account>:topicfilter/*/LastWillAndTestament",
        "arn:aws:iot:<region>:<account>:topicfilter/$aws/things/${iot:ClientId}/shadow/*",
        "arn:aws:iot:<region>:<account>:topicfilter/$aws/things/${iot:ClientId}/jobs/*"
      ]
    }
  ]
}

This policy only allows the CI to interact with topics that contain a matching client identifier (which is prefixed with the Travis CI variable IOT_IDENTIFIER_PREFIX). A few special topics used by the MQTT tests are also present. The policy will need to be updated if any new topics are used by the tests in the future.

Additional Travis CI setup for Jobs and Provisioning system tests

Additional CI settings for running the Jobs system tests and Provisioning system tests.

The Jobs system tests use AWS CLI to create and delete Jobs. Therefore, additional setup is needed in Travis CI.

Precondition

See Setting up Jobs system tests first to set up an AWS account for the Jobs system tests. Note that the coverage tests require an additional Thing to be registered (but can reuse the same AWS credentials).

Under the "Environment Variables" section of the Travis CI "Settings" page, set the following additional environment variables.

Name	Value	Notes
AWS_ACCESS_KEY_ID	The AWS access key for the Jobs tests IAM user	Attention This value should be hidden in the build log!
AWS_DEFAULT_REGION	The AWS region to use with the Jobs tests	Should be the region where the Jobs test Thing is registered. This value does not need to be hidden in the build log.
AWS_SECRET_ACCESS_KEY	The AWS secret access key for the Jobs tests IAM user	Attention This value should be hidden in the build log!