Observability

Getting Started

Copilot can configure trace collection for services by setting the following in a manifest:

observability:
  tracing: awsxray

For Request-Driven Web Services, Copilot will enable App Runner's baked-in tracing configuration.

For Load-Balanced Web Services, Backend Services, and Worker Services, Copilot will deploy the AWS OpenTelemetry Collector as a sidecar.

Instrumenting Your Service

Instrumenting your service to send telemetry data is done through language specific SDKs. Examples are provided in OpenTelemetry's documentation for each supported language. You can also view documentation and examples provided by AWS Distro for OpenTelemetry.

Example Application

This is a small Express.js service with instrumentation configured for all of its endpoints. To get started, install the required depedencies:

npm install express \
    @opentelemetry/api \
    @opentelemetry/sdk-trace-node \
    @opentelemetry/auto-instrumentations-node \
    @opentelemetry/exporter-trace-otlp-grpc \
    @opentelemetry/id-generator-aws-xray \
    @opentelemetry/propagator-aws-xray

Then, save the following to tracer.js:

const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { Resource } = require("@opentelemetry/resources");
const { trace } = require("@opentelemetry/api");
const { AWSXRayIdGenerator } = require("@opentelemetry/id-generator-aws-xray");
const { SemanticResourceAttributes } = require("@opentelemetry/semantic-conventions");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { AWSXRayPropagator } = require("@opentelemetry/propagator-aws-xray");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");

module.exports = (serviceName) => {
  const tracerConfig = {
    idGenerator: new AWSXRayIdGenerator(),
    instrumentations: [getNodeAutoInstrumentations()],
    resource: Resource.default().merge(
      new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: serviceName,
      })
    ),
  };

  const tracerProvider = new NodeTracerProvider(tracerConfig);
  const otlpExporter = new OTLPTraceExporter();

  tracerProvider.addSpanProcessor(new BatchSpanProcessor(otlpExporter));
  tracerProvider.register({
    propagator: new AWSXRayPropagator(),
  });

  return trace.getTracer("example-instrumentation");
};

tracer.js exports a function that returns a tracer configured to automatically export traces from an Express.js server using OpenTelemetry Protocol. We'll use this function in app.js, passing in a service name, copilot-observability.

'use strict';
const tracer = require('./tracer')('copilot-observability');
const app = require("express")();
const port = 8080;

app.get("/", (req, res) => {
    res.send("Hello World");
});

app.listen(port, () => {
    console.log(`Listening for requests on http://localhost:${port}`);
});

Now, if you deploy this service using Copilot and enable observability in the manifest, you'll be able to see traces generated by this service!

Including Trace Logs

Attention

This section is not applicable to Request-Driven Web Services

Because Copilot configures the ECS resource detector on the collector, traces generated by your service will include the log group your service is logging to. If you include the trace ID in your logs, logs associated with a trace will show up along with the trace in X-Ray. This can be helpful for understanding and debugging a trace.

X-Ray formats their trace IDs a little differently than OpenTelemetry, so you'll need a function like this to take an OpenTelemetry trace ID and format it for X-Ray:

function getXRayTraceId(span) {
    const id = span.spanContext().traceId;
    if (id.length < 9) {
        return id;
    }

    return "1-" + id.substring(0, 8) + "-" + id.substring(8);
}

Then, you can include the X-Ray trace ID in your logs like so:

console.log("[%s] A useful log message", getXRayTraceId(span));

Viewing Traces in CloudWatch

After you have instrumented your service and deployed it using Copilot, you're ready to view traces from your service! Depending on how you've instrumented your service, you will likely need to send a few requests to it before traces will appear in the AWS console.

First, open the CloudWatch console, and click on X-Ray traces/Service map in the menu. Here you can see a visual map of services interacting:

Next, you can view details of a specific trace by clicking on X-Ray traces/Traces in the menu and selecting a trace from the list.

In this example, you can see a service, js-copilot-observability, running some internal Express.js middleware, and then using the AWS SDK for Javascript to call s3:listBuckets: