Lambda Functions
Note: This documentation is also available in a rendered format here.
Deploys Lambda functions for data operations with VPC binding, EventBridge triggers (S3 notifications and scheduled rules), encrypted DLQ, Lambda layers, and Docker build support for complex dependencies. Common scenarios include running lightweight data transformations, responding to S3 upload events, executing scheduled data processing tasks, or integrating with external APIs as part of a data pipeline.
Deployed Resources
This module deploys and integrates the following resources:
Lambda Layers - Lambda layers which can be used in Lambda functions (inside or outside of this config)
Lambda Functions - Lambda functions for use in DataOps
- May be optionally VPC bound with configurable VPC, Subnet, and Security Group Parameters
- Can use an existing security group (from Project, for instance), or create a new security group per function
- DLQ automatically added for each Lambda with configurable retry/retention parameters
EventBridge Rules - EventBridge rules for triggering Lambda functions with events such as S3 Object Created Events
- EventBridge Notifications must be enabled on any bucket for which a rule is specified

Related Modules
- DataOps Project — Deploy the shared project infrastructure (KMS keys, security groups) that Lambda functions reference
- Step Functions — Orchestrate Lambda functions with Step Functions state machines
- Dashboard — Visualize Lambda function metrics and logs in CloudWatch dashboards
- EventBridge — Deploy custom event buses that Lambda functions can publish to or be triggered by
- Data Lake — Lambda functions can process data in data lake S3 buckets via EventBridge S3 notifications
Security/Compliance Details
This module is designed in alignment with MDAA security/compliance principles and CDK nag rulesets. Additional review is recommended prior to production deployment, to assist in meeting organization-specific compliance requirements.
- Encryption at Rest:
- Function environment variables encrypted with project KMS key
- DLQ messages encrypted with project KMS key
- Least Privilege:
- Execution roles specified per function
- Configurable reserved concurrency to prevent resource exhaustion
- Network Isolation:
- Optional VPC binding with configurable egress rules (CIDR, security group, prefix list)
- Per-function security groups deny all ingress by default
- All egress allowed by default (configurable)
AWS Service Endpoints
The following VPC endpoints may be required for VPC-bound Lambda functions if public AWS service endpoint connectivity is unavailable (e.g., private subnets without NAT gateway, firewalled environments, or PrivateLink-only architectures):
| AWS Service | Endpoint Service Name | Type |
|---|---|---|
| Lambda | com.amazonaws.{region}.lambda |
Interface |
| KMS | com.amazonaws.{region}.kms |
Interface |
| S3 | com.amazonaws.{region}.s3 |
Gateway |
| SQS | com.amazonaws.{region}.sqs |
Interface |
| CloudWatch Logs | com.amazonaws.{region}.logs |
Interface |
| STS | com.amazonaws.{region}.sts |
Interface |
| SSM Parameter Store | com.amazonaws.{region}.ssm |
Interface |
| EventBridge | com.amazonaws.{region}.events |
Interface |
Additional VPC endpoints may be required depending on the AWS services accessed by your custom Lambda function code.
Configuration
MDAA Config
Add the following snippet to your mdaa.yaml under the modules: section of a domain/env in order to use this module:
dataops-lambda: # Module Name can be customized
module_path: '@aws-mdaa/dataops-lambda' # Must match module NPM package name
module_configs:
- ./dataops-lambda.yaml # Filename/path can be customized
Module Config Samples and Variants
Copy the contents of the relevant sample config below into the ./dataops-lambda.yaml file referenced in the MDAA config snippet above.
Minimal Configuration
Deploys a single Lambda function with project autowiring. Start here for a basic data operations function within an existing DataOps project.
# Contents available via above link
# Minimal DataOps Lambda module configuration.
# Deploys a single Lambda function with project autowiring.
# (Optional) DataOps project name for Lambda resource autowiring.
projectName: dataops-project-test
# (Optional) Lambda function definitions.
functions:
- functionName: my-function
# Function source code directory
srcDir: ./src/lambda/test
# Code path to the Lambda handler function.
handler: test.lambda_handler
# The runtime for the function source code.
runtime: python3.13
# The role with which the Lambda function will be executed.
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# Number of times Lambda (0-2) will retry.
retryAttempts: 2
# Max age of an invocation event in seconds.
maxEventAgeSeconds: 3600
Comprehensive Configuration
Demonstrates Lambda functions and layers with VPC connectivity, environment variables, event schedules, and SQS triggers, all wired to a DataOps project. Start here when evaluating all available options for VPC binding, event triggers, layers, and concurrency settings.
sample-config-comprehensive.yaml
# Contents available via above link
# Sample config for the DataOps Lambda module - project variant.
# Demonstrates Lambda functions and layers with VPC connectivity,
# environment variables, event schedules, and SQS triggers, all
# wired to a DataOps project.
# (Optional) DataOps project name for Lambda function resource
# autowiring.
projectName: dataops-project-test
# (Optional) SNS topic ARN for job notifications and workflow alerts.
# Auto-resolved from project when projectName is set.
notificationTopicArn: arn:{{partition}}:sns:{{region}}:{{account}}:test-topic
# (Optional) Lambda layer definitions for shared code and
# dependencies across functions.
layers:
- layerName: test-layer
src: ./src/lambda/test
description: 'test layer'
# (Optional) Lambda function definitions for serverless data
# processing within the project.
functions:
# Required function parameters
- functionName: testfun # Function name. Must be unique within the config.
# (Optional) Function Description
description: Function descriptions
# Function source code directory
srcDir: ./src/lambda/test
# Code path to the Lambda handler function.
handler: test.lambda_handler
# The runtime for the function source code.
runtime: python3.13
# The role with which the Lambda function will be executed
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# Number of times Lambda (0-2) will retry before the invocation event
# is sent to DLQ.
retryAttempts: 2
# The max age of an invocation event before it is sent to DLQ, either due to
# failure, or insufficient Lambda capacity.
maxEventAgeSeconds: 3600
# (Optional) Number of seconds after which the function will time out.
# Default is 3 seconds
timeoutSeconds: 10
# (Optional) Set of environment variables and values to be passed to function
environment:
env-var-name: value
# (Optional) Number of reserved concurrent instances to be configured on the function.
# Ensures function always has this amount of concurrency available, but
# is subtracted from the overall account-wide concurrency limits.
# Default is to not reserve concurrency, and use the account-wide pool.
reservedConcurrentExecutions: 100
# (Optional) Size of function execution memory in MB
# Default is 128MB
memorySizeMB: 512
# (Optional) Size of function ephemeral storage in MB
# Default is 1024MB
ephemeralStorageSizeMB: 1024
# (Optional) Principal ARN granted Lambda invoke permissions.
grantInvoke: 'arn:{{partition}}:iam::{{account}}:role/invoker-role'
# (Optional) Additional resource permissions mapped by SID.
additionalResourcePermissions:
AllowS3Invoke:
# Lambda action (e.g., lambda:InvokeFunction).
action: 'lambda:InvokeFunction'
# AWS principal ARN for Lambda function access.
principal: 's3.amazonaws.com'
# Optional source account restriction for cross-account security.
sourceAccount: '{{account}}'
# Optional source resource ARN restriction for fine-grained access control.
sourceArn: 'arn:{{partition}}:s3:::my-trigger-bucket'
# Integration with Event Bridge for the purpose
# of triggering this function with Event Bridge rules
eventBridge:
# Number of times Event Bridge will attempt to trigger this function
# before sending event to DLQ. Note that Event Bridge Lambda invocation
# is async, so Lambda Function execution errors will generally be handled
# on the Lambda side itself.
retryAttempts: 10
# The max age of an event before Event Bridges sends it to DLQ.
maxEventAgeSeconds: 3600
# List of s3 buckets and prefixes which will be monitored via EventBridge in order to trigger this function
# Note that the S3 Bucket must have Event Bridge Notifications enabled.
s3EventBridgeRules:
testing-event-bridge-s3:
# The bucket producing event notifications
buckets: [sample-org-dev-instance1-datalake-raw]
# Optional - The S3 prefix to match events on
prefixes: [data/test-lambda/]
# Optional - Can specify a custom event bus for S3 rules, but note that S3 EventBridge notifications
# are initially sent only to the default bus in the account, and would need to be
# forwarded to the custom bus before this rule would match.
eventBusArn: 'arn:{{partition}}:events:{{region}}:{{account}}:event-bus/some-custom-name'
# List of generic Event Bridge rules which will trigger this function
eventBridgeRules:
testing-event-bridge:
description: 'testing'
eventBusArn: 'arn:{{partition}}:events:{{region}}:{{account}}:event-bus/some-custom-name'
eventPattern:
source:
- 'glue.amazonaws.com'
detail:
some_event_key: some_event_value
testing-event-bridge-schedule:
description: 'testing'
# (Optional) - Rules can be scheduled using a crontab expression
scheduleExpression: 'cron(0 20 * * ? *)'
# (Optional) - If specified, this input will be passed as the event payload to the function.
# If not specified, the matched event payload will be passed as input.
input:
some-test-input-obj:
some-test-input-key: test-value
# Example of a function which is VPC bound using a custom security group for this function
- functionName: testfunvpc
srcDir: ./src/lambda/test
handler: test.lambda_handler
runtime: python3.13
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# If vpcConfig is specified, Lambda will be VPC bound
vpcConfig:
# Required - The VPC on which the lambda will be bound
vpcId: 'some-vpc-id'
# Required - the list of subnet ids on which ENIs will be created for the Lambda
subnetIds:
- 'some-subnet-id'
# Optional - If specified, custom security group egress rules will be generated, and
# outgoing traffic from the function will be limited to these rules.
# If not specified, all outgoing traffic from the function will be permitted.
securityGroupEgressRules:
# Allow egress to a CIDR range
ipv4:
- cidr: 10.0.0.0/8
port: 443
protocol: tcp
# Optional description for the rule
description: 'Allow HTTPS to internal network'
# CIDR rule with port range (toPort defines upper bound)
- cidr: 172.16.0.0/12
port: 8080
# Ending port number defining the upper bound of the port range
toPort: 8090
protocol: tcp
description: 'Allow custom port range to private network'
# Allow egress to another Security Group
sg:
- sgId: sg-12312412412
port: 443
protocol: tcp
description: 'Allow HTTPS to peer security group'
# Ending port number defining the upper bound of the port range
toPort: 443
# Allow egress to a prefixlist
prefixList:
- prefixList: some-prefixlist-id
port: 443
protocol: tcp
description: 'Allow HTTPS via prefix list'
# Ending port number defining the upper bound of the port range
toPort: 443
# Example of a function which is VPC bound using an existing security group
- functionName: testfunvpcexistingsg
srcDir: ./src/lambda/test
handler: test.lambda_handler
runtime: python3.13
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# If vpcConfig is specified, Lambda will be VPC bound
vpcConfig:
# Required - The VPC on which the lambda will be bound
vpcId: 'some-vpc-id'
# Required - the list of subnet ids on which ENIs will be created for the Lambda
subnetIds:
- 'some-subnet-id'
# Optional - If specified, this security group will be bound to the Lambda function vpc interfaces
# In this example, we are using a security group generated by the DataOps Project module
securityGroupId: project:securityGroupId/test-security-group
# Example of a function which uses Lambda layers
- functionName: testlayerfunction
srcDir: ./src/lambda/test
handler: test.lambda_handler
runtime: python3.13
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
layerArns:
some-existing-layer: some-existing-layer-arn
generatedLayerNames:
- 'test-layer'
# Example of a function which uses a DockerBuild
- functionName: testdockerfunction
srcDir: ./src/lambda/docker
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# If true, lambda function will be built and deployed using Docker
# In this case, the srcDir is expected to container a Dockerfile
dockerBuild: true
# Example of a function with CloudWatch Logs Insights queries
- functionName: testobservabilityfunction
srcDir: ./src/lambda/test
handler: test.lambda_handler
runtime: python3.13
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
description: Function with CloudWatch observability features
# CloudWatch Logs Insights queries for analyzing function logs
logInsightsQueries:
# Query without explicit log groups (will use function's log group)
- queryName: testobservabilityfunction/errors
queryString: |
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
# Query with explicit log groups for cross-function analysis
- queryName: testobservabilityfunction/performance
queryString: |
fields @timestamp, @duration, @billedDuration, @memorySize, @maxMemoryUsed
| stats avg(@duration), max(@duration), min(@duration), avg(@maxMemoryUsed)
logGroupNames:
- /aws/lambda/testobservabilityfunction
- /aws/lambda/testfun
# Query without limit clause (will be auto-added)
- queryName: testobservabilityfunction/recent-invocations
queryString: |
fields @timestamp, @requestId, @message
| sort @timestamp desc
# CloudWatch metric filters for extracting custom metrics from logs
metricFilters:
# Filter with single transformation
- filterName: error-count
filterPattern: '{ $.level = "error" }'
metricTransformations:
- metricName: error-count
metricNamespace: TestObservability/Errors
metricValue: '1'
unit: Count
# Default value when filter pattern does not match.
defaultValue: 0
# Filter with multiple transformations
- filterName: processing-metrics
filterPattern: '[timestamp, request_id, level, msg, duration_ms, memory_mb]'
metricTransformations:
- metricName: processing-duration-ms
metricNamespace: TestObservability/Performance
metricValue: '$duration_ms'
unit: Milliseconds
- metricName: memory-usage-mb
metricNamespace: TestObservability/Performance
metricValue: '$memory_mb'
unit: Megabytes
# Filter with dimensions
- filterName: error-count-with-dimensions
filterPattern: '{ $.level = "error" }'
metricTransformations:
- metricName: error-count-by-type
metricNamespace: TestObservability/Errors
metricValue: '1'
unit: Count
dimensions:
Environment: test
Service: observability
# CloudWatch alarms for monitoring metrics
alarms:
# Simple single metric alarm referencing custom metric
- alarmName: high-error-rate
metricName: error-count
namespace: TestObservability/Errors
statistic: Sum
period: 300
evaluationPeriods: 3
# Datapoints that must breach threshold (M out of N evaluation).
datapointsToAlarm: 2
threshold: 5
comparisonOperator: GreaterThanOrEqualToThreshold
treatMissingData: notBreaching
alarmDescription: Alert when error count exceeds threshold
actionsEnabled: true
# CloudWatch metric unit.
unit: Count
alarmActions:
- project:projectTopicArn/default
# AWS Lambda metric alarm with {{functionName}} placeholder
- alarmName: lambda-errors
metricName: Errors
namespace: AWS/Lambda
statistic: Sum
period: 60
evaluationPeriods: 1
threshold: 1
comparisonOperator: GreaterThanOrEqualToThreshold
alarmDescription: Alert on Lambda function errors
dimensions:
FunctionName: '{{functionName}}'
alarmActions:
- 'arn:{{partition}}:sns:{{region}}:{{account}}:test-alerts'
okActions:
- 'arn:{{partition}}:sns:{{region}}:{{account}}:test-alerts'
# Metric math alarm (advanced)
- alarmName: total-errors-across-metrics
evaluationPeriods: 1
threshold: 10
comparisonOperator: GreaterThanOrEqualToThreshold
treatMissingData: notBreaching
alarmDescription: Alert when total errors across all metrics exceed threshold
metrics:
- id: total
expression: 'm1+m2'
label: 'Total Errors'
returnData: true
- id: m1
metricName: error-count
namespace: TestObservability/Errors
statistic: Sum
period: 300
- id: m2
metricName: error-count-by-type
namespace: TestObservability/Errors
statistic: Sum
period: 300
# Metric dimensions for filtering to specific instances.
dimensions:
Environment: test
# CloudWatch metric unit (e.g., Count, Milliseconds).
unit: Count
alarmActions:
- 'arn:{{partition}}:sns:{{region}}:{{account}}:test-alerts'
insufficientDataActions:
- 'arn:{{partition}}:sns:{{region}}:{{account}}:test-alerts'
Standalone Configuration (No Project)
Demonstrates standalone Lambda functions and layers with explicit KMS, bucket, deployment role, and security configuration. Use this when deploying outside of a DataOps project, providing infrastructure references directly.
# Contents available via above link
# Sample config for the DataOps Lambda module - no-project variant.
# Demonstrates standalone Lambda functions and layers with explicit
# KMS, bucket, deployment role, and security configuration.
# (Optional) KMS key ARN for encrypting DataOps resources and data.
# Auto-resolved from project when projectName is set.
kmsArn: arn:{{partition}}:kms:{{region}}:{{account}}:key/test-key-id
# (Optional) S3 bucket name for project storage (scripts, artifacts,
# temp files). Auto-resolved from project when projectName is set.
bucketName: test-lambda-bucket
# (Optional) IAM role ARN for deployment operations and resource
# management. Auto-resolved from project when projectName is set.
deploymentRoleArn: arn:{{partition}}:iam::{{account}}:role/test-deploy-role
# (Optional) Glue security configuration name for job encryption
# (at rest, in transit, CloudWatch logs). Auto-resolved from project
# when projectName is set.
securityConfigurationName: test-security-config
# (Optional) SNS topic ARN for job notifications and workflow alerts.
# Auto-resolved from project when projectName is set.
notificationTopicArn: arn:{{partition}}:sns:{{region}}:{{account}}:test-topic
# (Optional) Lambda layer definitions for shared code and
# dependencies across functions.
layers:
- layerName: test-layer
src: ./src/lambda/test
description: 'test layer'
# (Optional) Lambda function definitions for serverless data
# processing within the project.
functions:
# Required function parameters
- functionName: testfun # Function name. Must be unique within the config.
# (Optional) Function Description
description: Function descriptions
# Function source code directory
srcDir: ./src/lambda/test
# Code path to the Lambda handler function.
handler: test.lambda_handler
# The runtime for the function source code.
runtime: python3.13
# The role with which the Lambda function will be executed
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# Number of times Lambda (0-2) will retry before the invocation event
# is sent to DLQ.
retryAttempts: 2
# The max age of an invocation event before it is sent to DLQ, either due to
# failure, or insufficient Lambda capacity.
maxEventAgeSeconds: 3600
# (Optional) Number of seconds after which the function will timeout.
# Default is 3 seconds
timeoutSeconds: 10
# (Optional) Set of environment variables and values to be passed to function
environment:
env-var-name: value
# (Optional) Number of reserved concurrent instances to be configured on the function.
# Ensures function always has this amount of concurrency available, but
# is subtracted from the overall account-wide concurrency limits.
# Default is to not reserve concurrency, and use the account-wide pool.
reservedConcurrentExecutions: 100
# (Optional) Size of function execution memory in MB
# Default is 128MB
memorySizeMB: 512
# (Optional) Size of function ephemeral storage in MB
# Default is 1024MB
ephemeralStorageSizeMB: 1024
# (Optional) Principal ARN granted Lambda invoke permissions.
grantInvoke: 'arn:{{partition}}:iam::{{account}}:role/invoker-role'
# (Optional) Additional resource permissions mapped by SID.
additionalResourcePermissions:
AllowS3Invoke:
# Lambda action (e.g., lambda:InvokeFunction).
action: 'lambda:InvokeFunction'
# AWS principal ARN for Lambda function access.
principal: 's3.amazonaws.com'
# Optional source account restriction for cross-account security.
sourceAccount: '{{account}}'
# Optional source resource ARN restriction for fine-grained access control.
sourceArn: 'arn:{{partition}}:s3:::my-trigger-bucket'
# Integration with Event Bridge for the purpose
# of triggering this function with Event Bridge rules
eventBridge:
# Number of times Event Bridge will attempt to trigger this function
# before sending event to DLQ. Note that Event Bridge Lambda invocation
# is async, so Lambda Function execution errors will generally be handled
# on the Lambda side itself.
retryAttempts: 10
# The max age of an event before Event Bridges sends it to DLQ.
maxEventAgeSeconds: 3600
# List of s3 buckets and prefixes which will be monitored via EventBridge in order to trigger this function
# Note that the S3 Bucket must have Event Bridge Notifications enabled.
s3EventBridgeRules:
testing-event-bridge-s3:
# The bucket producing event notifications
buckets: [sample-org-dev-instance1-datalake-raw]
# Optional - The S3 prefix to match events on
prefixes: [data/test-lambda/]
# Optional - Can specify a custom event bus for S3 rules, but note that S3 EventBridge notifications
# are initially sent only to the default bus in the account, and would need to be
# forwarded to the custom bus before this rule would match.
eventBusArn: 'arn:{{partition}}:events:{{region}}:{{account}}:event-bus/some-custom-name'
# List of generic Event Bridge rules which will trigger this function
eventBridgeRules:
testing-event-bridge:
description: 'testing'
eventBusArn: 'arn:{{partition}}:events:{{region}}:{{account}}:event-bus/some-custom-name'
eventPattern:
source:
- 'glue.amazonaws.com'
detail:
some_event_key: some_event_value
testing-event-bridge-schedule:
description: 'testing'
# (Optional) - Rules can be scheduled using a crontab expression
scheduleExpression: 'cron(0 20 * * ? *)'
# (Optional) - If specified, this input will be passed as the event payload to the function.
# If not specified, the matched event payload will be passed as input.
input:
some-test-input-obj:
some-test-input-key: test-value
# Example of a function which is VPC bound using a custom security group for this function
- functionName: testfunvpc
srcDir: ./src/lambda/test
handler: test.lambda_handler
runtime: python3.13
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# If vpcConfig is specified, Lambda will be VPC bound
vpcConfig:
# Required - The VPC on which the lambda will be bound
vpcId: 'some-vpc-id'
# Required - the list of subnet ids on which ENIs will be created for the Lambda
subnetIds:
- 'some-subnet-id'
# Optional - If specified, custom security group egress rules will be generated, and
# outgoing traffic from the function will be limited to these rules.
# If not specified, all outgoing traffic from the function will be permitted.
securityGroupEgressRules:
# Allow egress to a CIDR range
ipv4:
- cidr: 10.0.0.0/8
port: 443
protocol: tcp
# Allow egress to another Security Group
sg:
- sgId: sg-12312412412
port: 443
protocol: tcp
# Allow egress to a prefixlist
prefixList:
- prefixList: some-prefixlist-id
port: 443
protocol: tcp
# Example of a function which is VPC bound using an existing security group
- functionName: testfunvpcexistingsg
srcDir: ./src/lambda/test
handler: test.lambda_handler
runtime: python3.13
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# If vpcConfig is specified, Lambda will be VPC bound
vpcConfig:
# Required - The VPC on which the lambda will be bound
vpcId: 'some-vpc-id'
# Required - the list of subnet ids on which ENIs will be created for the Lambda
subnetIds:
- 'some-subnet-id'
# Optional - If specified, this security group will be bound to the Lambda function vpc interfaces
# In this example, we are using a security group generated by the DataOps Project module
securityGroupId: test-security-group-id
# Example of a function which uses Lambda layers
- functionName: testlayerfunction
srcDir: ./src/lambda/test
handler: test.lambda_handler
runtime: python3.13
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
layerArns:
some-existing-layer: some-existing-layer-arn
generatedLayerNames:
- 'test-layer'
# Example of a function which uses a DockerBuild
- functionName: testdockerfunction
srcDir: ./src/lambda/docker
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
# If true, lambda function will be built and deployed using Docker
# In this case, the srcDir is expected to container a Dockerfile
dockerBuild: true
# Example of a function with CloudWatch Logs Insights queries
- functionName: testobservabilityfunction
srcDir: ./src/lambda/test
handler: test.lambda_handler
runtime: python3.13
roleArn: ssm:/sample-org/instance1/generated-role/lambda/arn
description: Function with CloudWatch observability features
# CloudWatch Logs Insights queries for analyzing function logs
logInsightsQueries:
# Query without explicit log groups (will use function's log group)
- queryName: testobservabilityfunction/errors
queryString: |
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
# Query with explicit log groups for cross-function analysis
- queryName: testobservabilityfunction/performance
queryString: |
fields @timestamp, @duration, @billedDuration, @memorySize, @maxMemoryUsed
| stats avg(@duration), max(@duration), min(@duration), avg(@maxMemoryUsed)
logGroupNames:
- /aws/lambda/testobservabilityfunction
- /aws/lambda/testfun
# Query without limit clause (will be auto-added)
- queryName: testobservabilityfunction/recent-invocations
queryString: |
fields @timestamp, @requestId, @message
| sort @timestamp desc
# CloudWatch metric filters for extracting custom metrics from logs
metricFilters:
# Filter with single transformation
- filterName: error-count
filterPattern: '{ $.level = "error" }'
metricTransformations:
- metricName: error-count
metricNamespace: TestObservability/Errors
metricValue: '1'
unit: Count
# Filter with multiple transformations
- filterName: processing-metrics
filterPattern: '[timestamp, request_id, level, msg, duration_ms, memory_mb]'
metricTransformations:
- metricName: processing-duration-ms
metricNamespace: TestObservability/Performance
metricValue: '$duration_ms'
unit: Milliseconds
- metricName: memory-usage-mb
metricNamespace: TestObservability/Performance
metricValue: '$memory_mb'
unit: Megabytes
# Filter with dimensions
- filterName: error-count-with-dimensions
filterPattern: '{ $.level = "error" }'
metricTransformations:
- metricName: error-count-by-type
metricNamespace: TestObservability/Errors
metricValue: '1'
unit: Count
dimensions:
Environment: test
Service: observability
# CloudWatch alarms for monitoring metrics
alarms:
# Simple single metric alarm referencing custom metric
- alarmName: high-error-rate
metricName: error-count
namespace: TestObservability/Errors
statistic: Sum
period: 300
evaluationPeriods: 1
threshold: 5
comparisonOperator: GreaterThanOrEqualToThreshold
treatMissingData: notBreaching
alarmDescription: Alert when error count exceeds threshold
actionsEnabled: true
alarmActions:
- arn:{{partition}}:sns:{{region}}:{{account}}:topic/test-topic
# AWS Lambda metric alarm with {{functionName}} placeholder
- alarmName: lambda-errors
metricName: Errors
namespace: AWS/Lambda
statistic: Sum
period: 60
evaluationPeriods: 1
threshold: 1
comparisonOperator: GreaterThanOrEqualToThreshold
alarmDescription: Alert on Lambda function errors
dimensions:
FunctionName: '{{functionName}}'
alarmActions:
- 'arn:{{partition}}:sns:{{region}}:{{account}}:test-alerts'
okActions:
- 'arn:{{partition}}:sns:{{region}}:{{account}}:test-alerts'
# Metric math alarm (advanced)
- alarmName: total-errors-across-metrics
evaluationPeriods: 1
threshold: 10
comparisonOperator: GreaterThanOrEqualToThreshold
treatMissingData: notBreaching
alarmDescription: Alert when total errors across all metrics exceed threshold
metrics:
- id: total
expression: 'm1+m2'
label: 'Total Errors'
returnData: true
- id: m1
metricName: error-count
namespace: TestObservability/Errors
statistic: Sum
period: 300
- id: m2
metricName: error-count-by-type
namespace: TestObservability/Errors
statistic: Sum
period: 300
alarmActions:
- 'arn:{{partition}}:sns:{{region}}:{{account}}:test-alerts'
insufficientDataActions:
- 'arn:{{partition}}:sns:{{region}}:{{account}}:test-alerts'