Skip to main content

Auditing

Understanding who accessed your cluster, what data was read or written, and which API calls were made is a key requirement for security, compliance, and operational troubleshooting. Amazon EMR integrates with several AWS services and open-source tools to provide auditing capabilities at different layers.

Amazon S3 Log Archival​

Enable S3 logging when you create your cluster to ensure that logs are preserved after nodes terminate. S3 logging must be explicitly enabled at cluster launch. You can enable it through the Amazon EMR console, the AWS CLI, or the API. Clusters launched from the console have S3 logging enabled by default; clusters launched from the CLI or API require you to specify it manually.

When S3 logging is enabled, Amazon EMR pushes logs from all cluster nodes to the specified S3 location every five minutes. Starting with Amazon EMR 6.9.0, logs are also archived to S3 immediately before nodes are terminated during scale-down or cluster shutdown, ensuring the most recent log data is preserved.

Amazon EMR captures three categories of logs for S3 archival:

  • System logs Logs generated by EMR daemons and the EMR service itself (provisioning, instance state, bootstrap actions).

  • Application logs Logs from open-source frameworks installed on each node of the cluster (Hadoop, Spark, Hive, HBase, and others).

  • Container logs The stdout and stderr output from individual YARN application containers, along with container launch scripts and directory metadata.

You can encrypt log files stored in Amazon S3 with an AWS KMS customer managed key. This option is available with Amazon EMR version 5.30.0 and later, excluding version 6.0.0. Enabling KMS encryption requires granting kms:GenerateDataKey to the EC2 instance profile and kms:DescribeKey to the EMR role and adding both as key users on the KMS key.

Starting with Amazon EMR 7.11.0, you can also control the S3 upload behavior independently for each log category using the S3LoggingConfiguration feature. This feature is useful when you need to control where log data is stored for compliance or data residency reasons — for example, ensuring that application logs are only written to a bucket in a specific Region under your control.

For additional details, see Configure Amazon EMR cluster logging and debugging in the Amazon EMR Management Guide.

AWS CloudTrail​

AWS CloudTrail captures all API calls for Amazon EMR as events, including the caller's identity, source IP address, timestamp, and request parameters. CloudTrail is active by default on your AWS account and provides a searchable, immutable Event history of the past 90 days of management events at no charge. For a longer retention period, create a CloudTrail trail to deliver logs to an Amazon S3 bucket.

Amazon EMR logs two categories of events to CloudTrail:

  • Management events (control plane) All EMR API operations such as RunJobFlow, TerminateJobFlows, AddJobFlowSteps, and SetTerminationProtection are logged by default. For a list of the AWS EMR control plane operations that AWS EMR logs to CloudTrail, see the AWS EMR API Reference.

  • Data events (data plane) Provide information about the resource operations performed on or in a resource (for example, reading or writing to an Amazon S3 object). Data events are not logged by default and must be explicitly enabled on your trail or event data store. Additional charges apply.

To support compliance and forensic analysis, deliver CloudTrail logs to an Amazon S3 bucket and query them with Amazon Athena. For more information, see Logging AWS EMR API calls using AWS CloudTrail and Querying AWS CloudTrail logs in the respective AWS documentation.

Identity propagation with EMR Runtime Roles

When jobs are submitted using EMR Runtime Roles, CloudTrail records the runtime role as the actor, hiding the original submitter. To preserve end-user attribution, set PropagateSourceIdentity to true in the ApplicationScopedIAMRoleConfiguration of your security configuration. EMR will then propagate the source identity from the calling credentials to the runtime role session, making it visible in CloudTrail's sourceIdentity field. For more details, see Runtime Roles for Amzon EMR Steps.

Amazon CloudWatch​

Amazon EMR publishes cluster metrics to Amazon CloudWatch automatically at five-minute intervals at no additional charge. These metrics cover cluster status, HDFS utilization, YARN resource usage, and node health. Metrics are retained for 63 days. For more information, see Monitoring Amazon EMR metrics with CloudWatch in the Amazon EMR Management Guide.

Native CloudWatch Logs Integration (Amazon EMR 7.11.0+)​

Starting with Amazon EMR 7.11.0, Amazon EMR provides native integration with Amazon CloudWatch Logs. When enabled, EMR automatically streams cluster logs to CloudWatch as they are generated, providing near-real-time access to log data, searchability, configurable retention policies, and the ability to stream to downstream systems such as Amazon OpenSearch Service or third-party platforms. This feature is configured at cluster creation and is only available when you submit applications using an EMR Step.

EMR supports three log types for CloudWatch delivery:

  • STEP_LOGS EMR step execution logs, including step controller output.

  • SPARK_DRIVER Apache Spark driver logs from Spark applications.

  • SPARK_EXECUTOR Apache Spark executor logs from worker nodes.

Each log type captures both STDOUT and STDERR streams. By default, only STEP_LOGS and SPARK_DRIVER are enabled. Logs are organized into a CloudWatch log group (default: /aws/emr/{cluster_id}) with log streams named by EMR step ID or YARN container ID. You can specify a custom log group name and log stream prefix to match your organization's naming conventions. Logs can optionally be encrypted at rest with an AWS KMS key.

To use this feature, the EC2 instance profile for the cluster must have CloudWatch Logs permissions. For clusters in private subnets, a VPC endpoint for CloudWatch Logs is also required. For the full configuration reference and CLI examples, see Publish Amazon EMR logs to CloudWatch Logs in the Amazon EMR Management Guide.

CloudWatch Agent (Earlier Releases)​

For EMR releases prior to 7.11.0, you can install and configure the Amazon CloudWatch agent manually via a bootstrap action to push application and system logs to CloudWatch Logs. Point the agent at the Hadoop log directories under /mnt/var/log/. This approach provides the same CloudWatch Logs benefits but requires manual agent configuration rather than the native integration. For additional details and examples, see Enhanced Monitoring with CloudWatch using custom metrics and logs.