DataOps Project
Note: This documentation is also available in a rendered format here.
Deploys shared DataOps project infrastructure including KMS keys, S3 project buckets, Glue databases, Lake Formation grants, security configurations, security groups, Glue connections/classifiers, and optional DataZone/SageMaker project integration. Use this module as the foundation for any data operations project, providing the shared encryption, storage, networking, and catalog resources that other DataOps modules depend on.
Deployed Resources
This module deploys and integrates the following resources:
Project KMS Key - Customer-managed KMS key used to encrypt all project resources at rest.
Project S3 Bucket - Storage for project activities (scratch, temporary, scripts, artifacts). Used as temp location for all project Glue jobs and to deploy/stage Glue job code.
Glue Databases - Catalog databases for crawled/generated tables.
LakeFormation Grants - Data lake location and read/write permission grants for project roles, with optional cross-account resource links and tag-based access control. The createReadWriteGrantsForProjectExecutionRoles flag accepts the case-sensitive permission levels read, write, and super in addition to boolean values, where true is equivalent to write and super additionally grants table ALTER/DROP and database DROP. Select super only for execution roles whose ETL jobs must replace or recreate catalog objects; because ALTER/DROP are destructive privileges that expand the role's blast radius, prefer read or write (the least-privilege levels) by default and reserve super for roles that genuinely require it.
Project Glue Security Config - Encrypts all job output, logging, and bookmark data with the project KMS key.
Project Security Groups - Configurable security groups for Glue connections and other project resources.
Glue Connections - Network and JDBC connections for reuse across project jobs and crawlers.
Glue Custom Classifiers - Classifiers (CSV, Grok, JSON, XML) for reuse across project crawlers.
DataZone/SageMaker Project (Optional) - Registers DataOps project resources as DataZone/SageMaker project, data sources, and assets. On the SageMaker path, raw S3 locations (outside the Glue catalog) can also be registered as data sources via sagemaker.project.dataSources.<name>.s3. sagemaker.project.glossaries and metadataForms take the same fields as the SageMaker Project module (this module builds SagemakerProjectL3Construct under the hood) - see that module's Glossaries section for the config surface, including description/readme/relatesTo.
Subscription Role Authorization (Optional) - Created when subscriptionAuthorizedRoleGroups is set: a Lambda-backed custom resource (Custom::DataZoneProjectRoleTagger, SageMaker path) that tags the listed roles with the project ID, and a managed policy (subscription-catalog-key-access) granting the listed roles decrypt on the domain's Glue catalog keys. On the DataZone path the roles are added to the project's subscription target instead of being tagged. See Subscribed Asset Access for Project Roles.
Environment ID Resolution - For each sagemaker.project.environmentConfigs entry (SageMaker path), a Lambda-backed custom resource (Custom::EnvironmentIdResolver) resolves the environment's live ID and publishes it to an SSM parameter at <ssmPath>/project/dataops-env-<configKey>/environment-id on every deploy, for use with the optional environmentId property. Setting environmentId is required to rename a project (or make any other change) once its environments are already ACTIVE - DataZone rejects such updates addressed by configuration name. See RENAMING_PROJECTS.md for details.
SNS Failure Notification Topic - SNS topic for publishing DataOps pipeline failure events, with optional email subscriptions.

Related Modules
- ETL Jobs — Deploy Glue ETL jobs that use project KMS keys, security configs, and connections
- Crawlers — Deploy Glue Crawlers that use project security configuration and connections
- Lambda Functions — Deploy Lambda functions for data operations using project KMS keys and security groups
- Workflows — Orchestrate project crawlers and jobs with Glue Workflows
- Step Functions — Orchestrate project resources with Step Functions state machines
- DataBrew — Deploy DataBrew jobs using project KMS keys for data profiling and transformation
- DynamoDB — Deploy DynamoDB tables encrypted with the project KMS key
- DMS — Deploy DMS replication tasks using project KMS keys for data migration
- Data Quality — Deploy Glue Data Quality rulesets for project databases and tables
- Dashboard — Create CloudWatch dashboards aggregating metrics from project Lambda functions and resources
- NiFi — Deploy Apache NiFi clusters using project KMS keys for data flow management
- Data Lake — Deploy data lake buckets that project jobs can read from and write to
- Lake Formation Access Control — Manage Lake Formation grants for Glue resources created outside of the project
- Glue Catalog - Share a producer account's Glue Catalog and catalog key with consumer accounts, which subscriptions to tables in other accounts require
- SageMaker (Domain) — Integrate project resources as SageMaker project data sources
- Roles — Create IAM roles for data engineer, execution, and data admin access
Security/Compliance Details
This module is designed in alignment with MDAA security/compliance principles and CDK nag rulesets. Additional review is recommended prior to production deployment, ensuring organization-specific compliance requirements are met.
- Encryption at Rest:
- Project KMS key encrypts all project resources (S3 bucket, Glue security config, job outputs, logs, bookmarks)
- Optional separate S3 output KMS key for data lake integration
- Least Privilege:
- KMS key usage access granted to project data engineer and execution roles via key policy
- KMS key usage/admin access granted to data admin role via key policy
- Project bucket read/write access granted by prefix to data engineer, execution, and data admin roles
- JDBC connection credentials managed via Secrets Manager dynamic references
- Lake Formation grants with tag-based access control and per-database/table permissions
- Subscription role tagging (SageMaker path) can set the
AmazonDataZoneProjecttag only on the listed roles and only to this project's ID, and can remove it only from roles that hold this project's ID - Roles in the groups listed in
subscriptionAuthorizedRoleGroups(opt-in) receive the project's subscribed tables: through theAmazonDataZoneProjecttag on the SageMaker path, which also makes them Lake Formation members of the project, or as authorized principals of the subscription target on the DataZone path. They also getkms:DescribeKeyon the domain's Glue catalog keys, andkms:Decrypton them only through Glue (kms:ViaService), so they cannot decrypt Glue connection passwords directly. Roles referenced by a literal ARN in another account are left out. Roles markedssoorimmutable, or referenced by an IAM Identity Center ARN or role name, are never tagged and never get the catalog key policy, so they receive subscribed tables only on the DataZone path
- Separation of Duties:
- Role-based access at data engineer, execution, and data admin levels
- Cross-account resource links for multi-account data governance
- Network Isolation:
- Configurable security groups with self-referencing ingress (required by Glue)
- All egress permitted by default, all other ingress denied
- VPC and JDBC connections for private network access
Subscribed Asset Access for Project Roles
When a DataZone or SageMaker Unified Studio (SMUS) project subscribes to a Glue table, the table is granted to the project's own role only. Glue jobs and crawlers from other DataOps modules run under projectExecutionRoles, and engineers and admins query from Athena under their own roles, so none of them can read subscribed tables by default.
Set subscriptionAuthorizedRoleGroups under sagemaker or datazone to list the role groups that should receive the project's subscriptions: projectExecutionRoles, dataEngineerRoles, and dataAdminRoles. A role in more than one listed group is authorized once if every group references it the same way, or one group by name and another by literal ARN, so reference each role the same way in every group. Access follows the subscription workflow: a listed role is granted a table when a subscription is approved and loses it when the subscription is revoked, without a redeploy.
sagemaker:
domainConfigSSMParam: /sample-org/shared/sagemaker/domain/test-domain/config
subscriptionAuthorizedRoleGroups:
- projectExecutionRoles
- dataEngineerRoles
project:
profileName: test-profile
What a listed role receives
The tables the project subscribes to, plus decrypt access through Glue to the domain's Glue catalog keys, which reading a table from another account's catalog requires. Everyone who can assume a listed role gets this access, so list engineer and admin groups only when all of their users should read every subscribed table. For a role whose trust policy trusts the account root, that includes every SMUS project user role in the account, which SageMakerStudioProjectUserRolePolicy allows to assume any role, and on the SageMaker path this project's EMR instance role, which SageMakerStudioEMRInstanceRolePolicy allows to assume roles tagged for its project. Give listed roles trust policies that name specific principals.
- SageMaker path: each role is tagged
AmazonDataZoneProject=<projectId>. SMUS fulfills subscriptions with Lake Formation grants to the principals carrying the consumer project's ID in that tag, which includes subscriptions approved before the role was listed. The tag also makes the role a Lake Formation member of the project, with create, drop, and write on the project database. AWS documents that Lake Formation treats this tag key specially (ABAC considerations) but not that SMUS grants subscriptions by it, so treat the subscription grants as observed behavior. - Lake Formation enforcement (SageMaker path): Lake Formation treats every principal carrying the
AmazonDataZoneProjecttag as opted in to Lake Formation permissions for all Data Catalog resources (ABAC considerations). A listed role therefore stops receiving access throughIAMAllowedPrincipalsor hybrid access mode on every Data Catalog table, not only the project's, and needs Lake Formation permissions on each database and table it uses. Grant those before listing a role whose jobs, crawlers, or queries rely on IAM-only access. - DataZone path: each role is added to the authorized principals of the subscription target MDAA creates for the project. DataZone grants a subscribed table to the target's principals when it fulfills the subscription, so the role receives the subscriptions fulfilled after it is listed; AWS does not document that subscriptions fulfilled earlier are granted to it. Nor does AWS document whether removing a role from the target revokes the grants already made to it, so after unlisting a role, check its Lake Formation permissions and revoke any that remain.
Limits
- Roles in the project's account only. Grants reach only roles in the consumer project's own account; the subscribed tables can come from a producer in any account (see Related settings). A role referenced by a literal ARN in another account is left out with a warning; a role resolved at deploy time (for example
ssm-org:) cannot be checked at synth and must be in the project's account. - One project per role (SageMaker path). A role tag holds one value. Deployment fails, naming the role and the other project, if a listed role is already tagged for another project.
- IAM Identity Center roles (SageMaker path). IAM rejects changes to
AWSReservedSSO_*roles that do not come from IAM Identity Center, so they cannot be tagged. MDAA leaves out roles markedsso: trueorimmutable: true, and roles referenced by an Identity Center ARN or anAWSReservedSSO_*name, with a warning. Mark Identity Center rolessso: true; one referenced by a value resolved at deploy time fails the deployment when it is tagged. - Nine roles (DataZone path). A subscription target allows 10 authorized principals, one of which is the environment user role, so synth fails if the listed groups contain more than nine distinct role references. A role referenced in different ways (for example by
id:in one group and by name in another) counts, and is authorized, once per way. Roles markedsso: trueorimmutable: true, and roles referenced by an Identity Center ARN or anAWSReservedSSO_*name, are authorized but get no catalog key policy, because MDAA does not attach policies to them. - Adding roles to an existing target (DataZone path). The DataZone
UpdateSubscriptionTargetAPI reference says authorized principal updates are supported in batches of 5, so adding more than five roles to an existing project's target in one deployment may fail. If it does, add them over several deployments. - Removal (SageMaker path). Unlisting a role or destroying the module removes the tag, but only while it still holds this project's ID; MDAA can only remove the current project's tag. A role keeps another project's ID in two cases: a deployment replaces the SMUS project (so its ID changes) and unlists the role, which keeps the old project's ID; or a deployment replaces the project and lists the role, then rolls back, and the role keeps the ID of the project the rollback removes. The tagging handler logs a warning naming the role. Remove the tag manually before authorizing the role into another project.
- Recreated roles (SageMaker path). MDAA tags roles only when the listed roles or the project change, so a listed role that is deleted and recreated under the same name stays untagged until such a deployment.
Related settings
- Guard the tag (SageMaker path). Lake Formation matches the
AmazonDataZoneProjecttag on IAM roles and users and on session tags (attribute-based access control), and a session tag overrides a role tag with the same key. So anyone who can set the tag can authorize a principal into a project: withiam:TagRoleoriam:TagUser, by creating a role or user with the tag, or by passing it as a session tag when assuming a role whose trust policy allowssts:TagSession, including through SAML or OIDC federation and IAM Identity Center attributes for access control. Restrict the key, for example with an SCP that deniesiam:TagRole,iam:UntagRole,iam:TagUser,iam:UntagUser,iam:CreateRole,iam:CreateUser, andsts:TagSessionwhenForAnyValue:StringEqualsonaws:TagKeysincludesAmazonDataZoneProject. Exempt the MDAA deployment role, this module's tagging handler role, the roles SMUS uses to provision projects, and the SMUS project user roles (datazone_usr_role_*), which pass their own project ID as a session tag when they assume roles. An SCP does not cover federation, so also keep identity provider and Identity Center attribute mappings from passing the tag. - Glue 5.0 Spark SQL. Glue 5.0 jobs that read subscribed tables with native Spark SQL also need the Lake Formation application integration setting Allow external engines to access data in Amazon S3 locations with full table access.
- Producers in other accounts. SMUS can fulfill a subscription to a table in another account only if that account's Glue Catalog module lists the consumer account in
consumerAccountsorkmsKeyConsumerAccounts.
Configuration
MDAA Config
Add the following snippet to your mdaa.yaml under the modules: section of a domain/env in order to use this module:
dataops-project: # Module Name can be customized
module_path: '@aws-mdaa/dataops-project' # Must match module NPM package name
module_configs:
- ./dataops-project.yaml # Filename/path can be customized
Module Config Samples and Variants
Copy the contents of the relevant sample config below into the ./dataops-project.yaml file referenced in the MDAA config snippet above.
Minimal Configuration
Contains only the required property (dataAdminRoles) plus one database to demonstrate the core use case. Start here for a basic DataOps project with a single Glue database and admin role.
# Contents available via above link
# Minimal config for the DataOps Project module.
# Contains only the required property (dataAdminRoles) plus one
# database to demonstrate the core use case.
# See CONFIGURATION.md for role reference options (name, arn, id).
# Data admin roles with full administrative access to all project resources
dataAdminRoles:
- name: Admin
# (Optional) Data engineer roles with operational access to project resources
dataEngineerRoles:
- name: data-engineer
# (Optional) Pre-defined execution roles for project resource operations
projectExecutionRoles:
- name: glue-role
# (Optional) Glue database definitions for centralized metadata management
databases:
test-database:
description: Test Database
# S3 bucket name for database location
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
locationPrefix: data/test
Comprehensive Configuration
Covers all available configuration options using the SageMaker integration path. Start here when evaluating all available options for databases, connections, classifiers, Lake Formation grants, SageMaker integration, and S3 lifecycle rules for the project bucket (expiring transient temp/query data and aging cold data to cheaper storage classes).
sample-config-comprehensive.yaml
# Contents available via above link
# Comprehensive sample config for the DataOps Project module.
# Exercises ALL compatible non-excluded properties at full depth.
# Uses the SageMaker integration path (mutually exclusive with datazone).
# See CONFIGURATION.md for role reference options (name, arn, id).
# Data admin roles with full administrative access to all project resources.
# Roles can be referenced by name (auto-expanded to ARN) or by explicit ARN.
dataAdminRoles:
# Role by name (auto-expanded to ARN at deploy time)
- name: Admin
# Role by ARN
- arn: arn:{{partition}}:iam::{{account}}:role/sample-org-dev-instance1-roles-data-admin
# Role by MDAA-generated role ID
- id: generated-role-id:data-admin
# Data engineer roles with operational access to project resources (jobs, crawlers, databases).
dataEngineerRoles:
- arn: arn:{{partition}}:iam::{{account}}:role/sample-org-dev-instance1-roles-data-engineer
# Pre-defined execution roles for project resource operations (jobs, crawlers).
projectExecutionRoles:
- arn: ssm:/sample-org/instance1/generated-role/glue-role/arn
- id: generated-role-id:databrew
# Failure notification configuration for Glue job monitoring and alerting.
failureNotifications:
# Email addresses for failure notification delivery
email:
- user1@example.com
- user2@example.com
# Shared security group configurations for project resources.
securityGroupConfigs:
test-security-group:
# VPC ID for security group deployment
vpcId: test-vpcid
# Egress rules for outbound traffic control
securityGroupEgressRules:
# IPv4 CIDR block rules
ipv4:
- cidr: 10.10.10.0/24
protocol: TCP
port: 443
# Description of the rule
description: Allow HTTPS to internal network
# Ending port for port range
toPort: 443
# Security group peer rules
sg:
- sgId: sg-12312412123
protocol: TCP
port: 443
# Description of the SG rule
description: Allow HTTPS to peer SG
# Ending port for port range
toPort: 443
# Prefix list rules
prefixList:
- prefixList: pl-12345678
protocol: TCP
port: 443
# Description of the prefix list rule
description: Allow HTTPS via prefix list
# Ending port for port range
toPort: 443
# KMS key ARN for encrypting S3 output data from project operations.
s3OutputKmsKeyArn: ssm:/sample-org/instance1/datalake/kms/id
# KMS key ARN for Glue Catalog metadata encryption.
glueCatalogKmsKeyArn: ssm:/sample-org/shared/glue-catalog/kms/arn
# S3 lifecycle rules applied to the project bucket. Each rule targets a key prefix
# so transient data is expired and cold data is aged to cheaper storage without
# affecting primary project data. Rules here are sensible defaults; tune days,
# storage classes, and prefixes to your retention and cost requirements.
lifecycleConfiguration:
# Expire transient scratch data under temp/ and clean up failed uploads.
- # Unique lifecycle rule identifier
id: expire-temp
# Rule state (enum: Enabled, Disabled)
status: Enabled
# Key prefix the rule applies to
prefix: temp/
# Days after creation before current objects expire (deleted)
expirationdays: 7
# Days after which incomplete multipart uploads are aborted
abortIncompleteMultipartUploadAfter: 7
# Expire Athena query results, which are reproducible and short-lived.
- id: expire-athena-results
status: Enabled
prefix: athena-results/
expirationdays: 30
abortIncompleteMultipartUploadAfter: 7
# Age cold project data to cheaper storage and prune old noncurrent versions.
- id: archive-data
status: Enabled
prefix: data/
# Storage-class transitions for current object versions
transitions:
# Days after creation before transitioning (enum storageClass: STANDARD_IA,
# ONEZONE_IA, INTELLIGENT_TIERING, GLACIER_IR, GLACIER, DEEP_ARCHIVE)
- days: 90
storageClass: STANDARD_IA
- days: 365
storageClass: GLACIER
# Days after a version becomes noncurrent before it expires
noncurrentVersionExpirationDays: 90
# Number of newer noncurrent versions to retain before expiring older ones
noncurrentVersionsToRetain: 3
# Project-level Lake Formation configuration for centralized tag-based access control.
lakeFormation:
# Lake Formation tag definitions (key + allowed values) shared across all project databases
lfTags:
- tagKey: environment
tagValues: [dev, test, prod]
- tagKey: data_tier
tagValues: [bronze, silver, gold]
- tagKey: data_classification
tagValues: [public, internal, confidential]
# AWS account ID for tag catalog scope
catalogId: '{{account}}'
# Custom Glue classifier definitions for specialized data format recognition.
classifiers:
# CSV classifier
classifierCsv:
# Classifier type: 'csv', 'grok', 'json', or 'xml'
classifierType: 'csv'
# Format-specific classifier configuration properties
configuration:
csvClassifier:
# Allow recognition of single-column CSV files
allowSingleColumn: false
# Header detection: 'UNKNOWN', 'PRESENT', or 'ABSENT'
containsHeader: 'PRESENT'
# Field delimiter character
delimiter: '~'
# When true, disables automatic whitespace trimming
disableValueTrimming: false
# Explicit column names
header:
- columnA
- columnB
# Quote character for field enclosure
quoteSymbol: '^'
# Classifier name
name: my-csv-classifier
# Grok classifier
classifierGrok:
classifierType: 'grok'
configuration:
grokClassifier:
# Identifier of the data format
classification: special-logs
# Custom grok patterns
customPatterns: 'MESSAGEPREFIX .*-.*-.*-.*-.*'
# Grok pattern applied to data
grokPattern: '%{TIMESTAMP_ISO8601:timestamp} \[%{MESSAGEPREFIX:message_prefix}\] %{CRAWLERLOGLEVEL:loglevel} : %{GREEDYDATA:message}'
# Classifier name
name: my-grok-classifier
# JSON classifier
classifierJson:
classifierType: 'json'
configuration:
jsonClassifier:
# JsonPath string defining the JSON data
jsonPath: '$[*]'
# Classifier name
name: my-json-classifier
# XML classifier
classifierXml:
classifierType: 'xml'
configuration:
xmlClassifier:
# Identifier of the data format
classification: xml-data
# XML tag designating the element containing each record
rowTag: '<row item_a="A" item_b="B"></row>'
# Classifier name
name: my-xml-classifier
# Glue connection definitions for secure connectivity to external data sources.
connections:
# NETWORK connection
connectionVpc:
# Connection type: 'JDBC', 'KAFKA', 'MONGODB', or 'NETWORK'
connectionType: NETWORK
# Description of the connection's purpose
description: VPC Connection Example
# Criteria for automated connection selection in ETL jobs
matchCriteria:
- network-match
# VPC networking requirements
physicalConnectionRequirements:
# Availability zone for connection placement
availabilityZone: '{{region}}a'
# Subnet ID for connection VPC placement
subnetId: subnet-123abc456def
# Existing security group IDs
securityGroupIdList:
- sg-890abc123asc
# NETWORK connection using project security group
connectionVpcWithProjectSG:
connectionType: NETWORK
description: VPC Connection with Project SG
physicalConnectionRequirements:
availabilityZone: '{{region}}a'
subnetId: subnet-09ba402b76a346ffb
# Project-generated security group names
projectSecurityGroupNames:
- test-security-group
# JDBC connection
connectionJdbc:
connectionType: JDBC
# Key-value pairs for authentication and connection configuration
connectionProperties:
JDBC_CONNECTION_URL: 'jdbc:awsathena://AwsRegion=[REGION];UID=[ACCESS KEY];PWD=[SECRET KEY];S3OutputLocation=[LOCATION]'
JDBC_ENFORCE_SSL: true
description: JDBC Connection Example
physicalConnectionRequirements:
availabilityZone: '{{region}}a'
subnetId: subnet-123abc456def
securityGroupIdList:
- sg-890abc123asc
# KAFKA connection
connectionKafka:
connectionType: KAFKA
connectionProperties:
KAFKA_BOOTSTRAP_SERVERS: 'broker1:9092,broker2:9092'
KAFKA_SSL_ENABLED: true
description: Kafka Streaming Connection
matchCriteria:
- kafka-match
physicalConnectionRequirements:
availabilityZone: '{{region}}a'
subnetId: subnet-123abc456def
securityGroupIdList:
- sg-890abc123asc
# MONGODB connection
connectionMongodb:
connectionType: MONGODB
connectionProperties:
CONNECTION_URL: 'mongodb://host:27017/database'
description: MongoDB Connection Example
physicalConnectionRequirements:
availabilityZone: '{{region}}a'
subnetId: subnet-123abc456def
securityGroupIdList:
- sg-890abc123asc
# SageMaker project integration (mutually exclusive with datazone).
sagemaker:
# SSM parameter name containing domain configuration
domainConfigSSMParam: /sample-org/shared/sagemaker/domain/test-domain/config
# Auto-assign data admin roles as project owners
createDataAdminOwners: false
# (Optional) Role groups from this config that receive every table the SageMaker Unified Studio
# project subscribes to (enum: projectExecutionRoles, dataEngineerRoles, dataAdminRoles)
# (default: [])
subscriptionAuthorizedRoleGroups:
- projectExecutionRoles
- dataEngineerRoles
project:
# Name of the project profile to use
profileName: test-profile
# (Optional) Exact DataZone project name used verbatim, bypassing MDAA naming.
# Free-form business label for the SMUS project title; 1-64 chars matching
# ^[\w -]+$ (spaces and mixed case allowed). All other resources keep MDAA naming.
nameOverride: Fraud Analytics Sandbox
# Domain unit path for project placement
domainUnit: /some/domain/unit
# MDAA group config names with PROJECT_CONTRIBUTOR designation
groups:
contributor-group: group-config-name
# MDAA group config names with PROJECT_OWNER designation
ownerGroups:
owner-group: owner-group-config-name
# MDAA user config names with PROJECT_OWNER designation
ownerUsers:
owner-user: owner-user-config-name
# MDAA user config names with PROJECT_CONTRIBUTOR designation
users:
contributor-user: user-config-name
# Per-environment configuration overrides
environmentConfigs:
test-env:
parameters:
key1: value1
# (Optional) Apply this entry to a specific environment ID instead of looking it up by name
test-env-2:
environmentId: test-environment-id
parameters:
key2: value2
# (Optional) Default values for publishOnImport, enableSetting, schedule, recommendation,
# autoImportDataQualityResult, and filterExpressions, inherited by every Glue (databaseName)
# entry in dataSources below. A data source that sets one of these fields itself overrides
# the default for that field entirely (fields are not deep-merged); has no effect on an s3
# data source. See test-source below.
# Also applies, in this sagemaker (SMUS) mode, to this project's own database-backed
# data sources (databases.<db>.createSagemakerDatasource/createDatazoneDatasource below).
dataSourceDefaults:
# (Optional) Auto-publish imported assets to the project's catalog
publishOnImport: true
# (Optional) Enabled setting enum: ENABLED | DISABLED
enableSetting: ENABLED
# (Optional) Schedule the data source runs on
schedule:
# (Required) Cron expression for the run schedule
schedule: cron(0 6 * * ? *)
# (Optional) Timezone the schedule is evaluated in
timezone: UTC
# (Optional) Business name generation recommendation
recommendation:
# (Optional) Enable automatic business name generation
enableBusinessNameGeneration: true
# Data sources for Glue databases not created by this DataOps Project
dataSources:
test-source:
# Glue database name to use as the data source
databaseName: non-project-database
# (Optional) Overrides dataSourceDefaults.publishOnImport (true, above) for this
# source only; unlike databases.<db>.createSagemakerDatasource/createDatazoneDatasource
# sources, which have no per-field overrides and always take dataSourceDefaults as-is
publishOnImport: false
# (Optional) Whether to auto-import AWS Glue Data Quality results
autoImportDataQualityResult: false
# (Optional) Limits imported tables to those matching these filter expressions.
# Controls catalog import only, not access: Lake Formation still grants read on
# every table in the database regardless of these filters.
filterExpressions:
- type: INCLUDE
expression: dim_*
- type: EXCLUDE
expression: tmp_*
# (Optional) Metadata forms (declared under this project's metadataForms, below) to
# attach to every asset this data source imports
metadataForms:
DataOpsGovernanceForm:
content: '{"owner":"data-team"}'
test-s3-source:
# S3 location to register as a data source, mutually exclusive with databaseName
s3:
bucketName: test-data-bucket
prefix: raw/
# (Optional) Business glossaries owned by this project. A metadata form field's
# glossaryId may reference a declared glossary here as "<glossaryName>", or a
# declared term as "<glossaryName>.<termName>", resolved by MDAA to the
# glossary's deploy-time ID.
glossaries:
GovernanceGlossary:
description: Governance terminology for DataOps projects
status: ENABLED
terms:
DataOwner:
description: The team accountable for this project's data
# (Optional) Longer-form Markdown documentation, shown as the term's README in
# SageMaker Unified Studio
readme: The team or individual responsible for this project's data quality and access
status: ENABLED
ExecutiveDataOwner:
description: A data owner with executive sign-off authority for sensitive datasets
# (Optional) Other declared terms this term "is a type of"; a bare name
# references a term in this same glossary. An ExecutiveDataOwner is a more
# specific kind of DataOwner, with added sign-off authority.
relatesTo:
- DataOwner
# (Optional) Metadata form types owned by this project. MDAA assembles the
# Smithy model; the form name is used verbatim as the structure name.
metadataForms:
DataOpsGovernanceForm:
# (Optional) Description of the form type
description: Project governance metadata
# (Required) Field name -> FormFieldProps
fields:
owner:
# (Required) Smithy prelude type
type: String
# (Optional) Annotates @required
required: true
# (Optional) Description of the field, annotated via @documentation
description: Team or individual accountable for this dataset
# (Optional) "<glossaryName>.<termName>" resolves to the glossary's deploy-time ID
glossaryId: GovernanceGlossary.DataOwner
reviewedOn:
type: Timestamp
# Glue database definitions for centralized metadata management.
databases:
# Full-featured database with all Lake Formation options
test-database1:
# Description of the database's purpose
description: Test Database 1
# S3 bucket name for database data storage location
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
# S3 prefix for data organization within the bucket
locationPrefix: data/test1
# Lake Formation configuration for access control
lakeFormation:
# Auto-create super grants for data admin roles
createSuperGrantsForDataAdminRoles: true
# Auto-create read grants for data engineer roles
createReadGrantsForDataEngineerRoles: true
# Auto-create grants for project execution roles. Accepts a boolean or a
# permission level (read | write | super). The 'super' level grants
# ALTER/DROP for ETL jobs that replace or recreate tables.
createReadWriteGrantsForProjectExecutionRoles: super
# Target account numbers for cross-account resource link creation
createCrossAccountResourceLinkAccounts:
- '{{context:account-2}}'
# Custom name for cross-account resource links
createCrossAccountResourceLinkName: 'testing'
# Named Lake Formation grant configurations
grants:
# Read grant example
example_read_grant:
# Database permissions level: 'read', 'write', or 'super'
databasePermissions: read
# Table permissions level: 'read', 'write', or 'super'
tablePermissions: read
# Specific table names for targeted grant creation
tables:
- test-table
# Named principal references
principals:
principalA:
# IAM SAML IDP ARN
federationProviderArn: some-federation-provider-arn
# Federated username
federatedUser: some-user-name
principalB:
federationProviderArn: some-federation-provider-arn
# Federated group
federatedGroup: some-group-name
# Principal with role reference
principalC:
# IAM role reference
role:
name: some-role-name
# AWS account ID for cross-account principal resolution
account: '{{context:account-2}}'
# Direct principal ARN mapping
principalArns:
principalD: some-other-role-arn
# Write grant example
example_write_grant:
databasePermissions: write
tablePermissions: write
principalArns:
writer-role: arn:{{partition}}:iam::{{account}}:role/writer-role
# Super grant example
example_super_grant:
databasePermissions: super
tablePermissions: super
principalArns:
super-role: arn:{{partition}}:iam::{{account}}:role/super-role
# LF-Tag values to associate with this database
databaseTagValues:
- tagKey: environment
tagValues: [dev]
- tagKey: data_tier
tagValues: [bronze]
- tagKey: data_classification
tagValues: [public]
# Tag-based grant configurations
tagBasedGrants:
dev_access:
# Map of principal names to IAM ARNs
principalArns:
dev-role: arn:{{partition}}:iam::{{account}}:role/dev-data-user
# Lake Formation permissions to grant
permissions: [DESCRIBE, SELECT]
# Resource type scope: DATABASE or TABLE
resourceType: TABLE
# LF-Tag expression defining resource selection
lfTagExpression:
environment: [dev]
data_tier: [bronze, silver]
# Grant with DATABASE resource type and permissionsWithGrantOption
db_admin_access:
principalArns:
db-admin: arn:{{partition}}:iam::{{account}}:role/db-admin
permissions: [DESCRIBE, ALTER, CREATE_TABLE]
# Permissions that recipients can further grant
permissionsWithGrantOption: [DESCRIBE]
resourceType: DATABASE
lfTagExpression:
environment: [prod]
# Database with crawler configuration
test-database-crawler:
description: Test Database with Crawler
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
locationPrefix: data/test-crawler
# Auto-create Glue Crawler for this database
crawler:
# Crawler execution role (required)
role:
name: glue-crawler-role
# Custom classifier names to use
classifiers:
- classifierCsv
# Crawler configuration object
extraConfiguration:
Version: 1.0
Grouping:
TableGroupingPolicy: CombineCompatibleSchemas
# Recrawl behaviour
recrawlBehavior: CRAWL_NEW_FOLDERS_ONLY
# Crawler execution schedule
schedule:
# Cron expression for scheduling
scheduleExpression: 'cron(15 12 * * ? *)'
# Schema change policy
schemaChangePolicy:
# Update behavior: LOG or UPDATE_IN_DATABASE
updateBehavior: UPDATE_IN_DATABASE
# Delete behavior: LOG, DELETE_FROM_DATABASE, or DEPRECATE_IN_DATABASE
deleteBehavior: LOG
# Table name prefix
tablePrefix: 'crawled_'
# Condensed DB config
test-database2:
description: Test Database 2
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
locationPrefix: data/test2
lakeFormation:
createSuperGrantsForDataAdminRoles: true
createReadGrantsForDataEngineerRoles: true
createReadWriteGrantsForProjectExecutionRoles: true
grants:
example_condensed_read_grant:
principalArns:
principalA: arn:{{partition}}:iam::{{account}}:role/cross-account-role
# Verbatim DB Name Config
test-database4:
description: Test Database 4
# Use exact database name without applying naming conventions
verbatimName: true
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
locationPrefix: data/test4
# Iceberg Compliant DB Name Config
test-database5:
description: Test Database 5
# Replace hyphens with underscores for Apache Iceberg compatibility
icebergCompliantName: true
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
locationPrefix: data/test5
# SageMaker Project Data Source
test-database-sus:
description: Test SageMaker Database
# Auto-create SageMaker data sources for this database
createSagemakerDatasource: true
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
locationPrefix: data/test-sus
SageMaker Integration Configuration
Extends the primary configuration with SageMaker domain integration, project profiles, and data sources for SageMaker-governed data access. Choose this variant when your organization uses SageMaker Unified Studio for data governance and you want project resources automatically registered as SageMaker data sources.
# Contents available via above link
# Arns for IAM role which will be authoring code within the project
dataEngineerRoles:
- name: data-engineer
# See CONFIGURATION.md for role reference options (name, arn, id).
# Arns for IAM roles which will be provided to the projects's resources (IE bucket)
dataAdminRoles:
- name: Admin
projectExecutionRoles:
- name: glue-role
# (Optional) - Generate a SageMaker Project for this DataOps Project (comment if using datazone)
sagemaker:
# The SSM Parameter containing domain config details for a SageMaker Domain created by the MDAA SageMaker module
domainConfigSSMParam: /test-org/test-domain/sagemaker/domain/test-domain/config
project:
profileName: test-profile
# Data sources for Glue databases or S3 locations not created by this DataOps Project
dataSources:
test-s3-source:
# S3 location to register as a data source, mutually exclusive with databaseName
s3:
bucketName: test-data-bucket
prefix: raw/
databases:
# SageMaker Project Data Source (comment if using datazone)
test-database-sus:
description: Test SageMaker Database
# Creates a datasource in the SageMaker Project associated with this DataOps Project
createSagemakerDatasource: true
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
locationPrefix: data/test-sus
DataZone Integration Configuration
Uses Amazon DataZone for data governance and catalog management instead of SageMaker Unified Studio. Choose this variant when your organization uses DataZone for data discovery and access management.
# Contents available via above link
# DataZone variant sample config for the DataOps Project module.
# Exercises the DataZone integration path (mutually exclusive with sagemaker).
# Data engineer roles with operational access to project resources
dataEngineerRoles:
- name: data-engineer
# See CONFIGURATION.md for role reference options (name, arn, id).
# Data admin roles with full administrative access to all project resources
dataAdminRoles:
- name: Admin
# Pre-defined execution roles for project resource operations
projectExecutionRoles:
- name: glue-role
# DataZone configuration for data governance and catalog integration.
# Mutually exclusive with sagemaker.
datazone:
# SSM parameter name containing domain configuration
domainConfigSSMParam: /test-org/test-domain/test-datazone/domain/test-domain/config
# (Optional) Role groups from this config that receive the tables the DataZone project
# subscribes to (enum: projectExecutionRoles, dataEngineerRoles, dataAdminRoles) (default: [])
subscriptionAuthorizedRoleGroups:
- projectExecutionRoles
- dataAdminRoles
# DataZone project configuration
project:
# (Optional) Exact DataZone project name used verbatim, bypassing MDAA naming.
# Free-form business label for the project title; 1-64 chars matching
# ^[\w -]+$ (spaces and mixed case allowed). All other resources keep MDAA naming.
nameOverride: Fraud Analytics Sandbox
# Domain unit identifier for organizational hierarchy
domainUnit: test-domain-unit
# DataZone environment configuration for Lake Formation integration
environment:
# Lake Formation manage access role reference
lakeformationManageAccessRole:
name: lf-manage-access-role
# MDAA module group config names with PROJECT_CONTRIBUTOR designation
groups:
contributor-group: group-config-name
# MDAA module group config names with PROJECT_OWNER designation
ownerGroups:
owner-group: owner-group-config-name
# MDAA module user config names with PROJECT_OWNER designation
ownerUsers:
owner-user: owner-user-config-name
# MDAA module user config names with PROJECT_CONTRIBUTOR designation
users:
contributor-user: user-config-name
databases:
# DataZone Project Data Source
test-database-datazone:
description: Test DataZone Database
# Auto-create DataZone data sources for this database
createDatazoneDatasource: true
# Often created by the Data Lake module.
# Example SSM: ssm:/{{org}}/{{domain}}/<datalake_module_name>/bucket/<zone_name>/name
locationBucketName: some-bucket-name
locationPrefix: data/test-sus