SageMaker Pipeline

Note: This documentation is also available in a rendered format here.

Deploys a SageMaker Pipeline defined entirely in CDK and CloudFormation with no seed code required. The pipeline definition is specified as JSON configuration supporting Processing, Training, RegisterModel, CreateModel, Transform, and Condition steps. This provides a fully declarative, infrastructure-as-code approach to ML workflows. Use this module when you need a SageMaker Pipeline managed through YAML configuration without maintaining separate seed code repositories, or as an alternative to the SageMaker MLOps module's CodeCommit-based approach.

Deployed Resources

This module deploys and integrates the following resources:

SageMaker Pipeline - ML workflow pipeline with step definitions for processing, training, model registration, and batch transform.

AWS IAM Pipeline Execution Role - Execution role for the SageMaker Pipeline with permissions to run training jobs, processing jobs, and register models.

AWS KMS Key - Customer-managed encryption key for S3 model bucket, training job volumes, and processing job storage.

Amazon S3 Model Bucket - Stores model artifacts, processing outputs, and pipeline step data.

SageMaker Model Package Group (Optional) - Registry for versioned model packages produced by RegisterModel pipeline steps.

AWS SSM Parameters - Publishes pipeline ARN, model bucket name, and model package group ARN for cross-module integration.

SageMaker MLOps — Alternative approach that uses CodeCommit seed code and CodePipeline for training orchestration instead of declarative pipeline definitions
SageMaker Endpoint — Deploys real-time inference endpoints from model packages registered by this module's pipeline
SageMaker Model Monitoring — Monitors endpoints serving models produced by this module's pipeline for drift and quality degradation
SageMaker Studio Domain — Provides SageMaker domain tagging context for resource governance

Security/Compliance Details

This module is designed in alignment with MDAA security/compliance principles and CDK nag rulesets. Additional review is recommended prior to production deployment, ensuring organization-specific compliance requirements are met.

Encryption at Rest:
- S3 model bucket encrypted with customer-managed KMS key
- Training job storage volumes encrypted with KMS
- Processing job storage volumes encrypted with KMS
- Model artifacts encrypted at rest in the model registry
Encryption in Transit:
- All S3 access enforced over HTTPS via bucket policy
- Inter-container traffic encryption enabled for distributed training steps
Least Privilege:
- Pipeline execution role scoped to specific S3 paths, KMS key, and SageMaker actions
- Model package group access restricted to the pipeline execution role
- Cross-account model registry access uses scoped IAM policies
Network Isolation:
- Pipeline steps support VPC configuration with security groups and subnets
- Training and processing containers can be configured for network isolation

Configuration

MDAA Config

Add the following snippet to your mdaa.yaml under the modules: section of a domain/env in order to use this module:

sagemaker-pipeline: # Module Name can be customized
  module_path: '@aws-mdaa/sagemaker-pipeline' # Must match module NPM package name
  module_configs:
    - ./sagemaker-pipeline.yaml # Filename/path can be customized

Module Config Samples and Variants

Copy the contents of the relevant sample config below into the ./sagemaker-pipeline.yaml file referenced in the MDAA config snippet above.

Minimal Configuration

Start here for a simple pipeline with a single training step and model registration using default instance types.

sample-config-minimal.yaml

# Minimal config for the SageMaker Pipeline module.
# Contains only the required properties for a basic pipeline
# with a single processing step.

# SageMaker project name used in resource naming
projectName: test-pipeline-minimal

pipeline:
  # Pipeline steps — at least one step is required
  steps:
    - name: PreprocessData
      type: Processing
      processing:
        imageUri: "123456789012.dkr.ecr.us-east-1.amazonaws.com/preprocess:latest"
        instanceType: ml.m5.xlarge
        outputs:
          - outputName: output

Comprehensive Configuration

Use this as a reference when you need multi-step pipelines with processing, training, conditional branching, model registration, VPC isolation, and cross-account model registry access.

sample-config-comprehensive.yaml

# Comprehensive config for the SageMaker Pipeline module.
# Creates a SageMaker Pipeline via CfnPipeline (pure CDK) with
# preprocessing, training, evaluation, conditional model
# registration, VPC isolation, and cross-account model registry.

# SageMaker project name used in resource naming
projectName: test-pipeline-project

# (Optional) SageMaker domain ID for Studio integration tagging
# Often created by the SageMaker Studio Domain module.
# Example SSM: ssm:/{{org}}/{{domain}}/<sm_studio_domain_module_name>/domain-id
domainId: d-test123

# (Optional) SageMaker domain ARN for Studio integration tagging
# Often created by the SageMaker Studio Domain module.
# Example SSM: ssm:/{{org}}/{{domain}}/<sm_studio_domain_module_name>/domain-arn
domainArn: arn:{{partition}}:sagemaker:{{region}}:{{account}}:domain/d-test123

# (Optional) Model Package Group name for model registration
modelPackageGroupName: test-mpg

# (Optional) Pre-prod account ID for cross-account model
# registry access
preProdAccountId: '{{context:account-2}}'

# (Optional) Prod account ID for cross-account model
# registry access
prodAccountId: '{{context:account-3}}'

pipeline:
  # (Optional) Pipeline parameters — configurable at execution
  parameters:
    - name: ProcessingInstanceType
      type: String
      defaultValue: ml.m5.xlarge
    - name: TrainingInstanceType
      type: String
      defaultValue: ml.m5.xlarge
    - name: InputDataUrl
      type: String
      defaultValue: s3://test-bucket/dataset/data.csv

  # (Optional) Network configuration for pipeline steps
  networkConfig:
    enableNetworkIsolation: true
    encryptInterContainerTraffic: true
    # Often created by your VPC/networking stack.
    # Example SSM: ssm:/path/to/subnet/id
    subnetIds:
      - subnet-abc
      - subnet-def
    # Often created by your VPC/networking stack.
    # Example SSM: ssm:/path/to/security-group/id
    securityGroupIds:
      - sg-123

  # Pipeline steps — define the ML workflow
  steps:
    - name: PreprocessData
      type: Processing
      processing:
        imageUri: "123456789012.dkr.ecr.us-east-1.amazonaws.com/preprocess:latest"
        instanceType: ml.m5.xlarge
        scriptS3Uri: s3://test-bucket/scripts/preprocessing.py
        inputs:
          - inputName: input-data
            s3Uri: s3://test-bucket/dataset/data.csv
        outputs:
          - outputName: train
          - outputName: validation
          - outputName: test
        enableNetworkIsolation: false

    - name: TrainModel
      type: Training
      training:
        imageUri: "123456789012.dkr.ecr.us-east-1.amazonaws.com/train:latest"
        instanceType: ml.m5.xlarge
        hyperparameters:
          objective: "reg:linear"
          num_round: "50"
          max_depth: "5"
          eta: "0.2"
        inputChannels:
          - channelName: train
            stepOutput: "PreprocessData.train"
            contentType: text/csv
          - channelName: validation
            stepOutput: "PreprocessData.validation"
            contentType: text/csv
        outputPath: s3://test-bucket/output

    - name: EvaluateModel
      type: Processing
      processing:
        imageUri: "123456789012.dkr.ecr.us-east-1.amazonaws.com/train:latest"
        instanceType: ml.m5.xlarge
        scriptS3Uri: s3://test-bucket/scripts/evaluate.py
        inputs:
          - inputName: model
            stepOutput: "TrainModel.modelArtifacts"
          - inputName: test
            stepOutput: "PreprocessData.test"
        outputs:
          - outputName: evaluation
        propertyFiles:
          - propertyFileName: EvaluationReport
            outputName: evaluation
            filePath: evaluation.json

    - name: RegisterModel
      type: RegisterModel
      register:
        imageUri: "123456789012.dkr.ecr.us-east-1.amazonaws.com/inference:latest"
        modelDataStep: TrainModel
        modelPackageGroupName: test-mpg
        approvalStatus: PendingManualApproval
        contentTypes: ["text/csv"]
        responseTypes: ["text/csv"]
        inferenceInstanceTypes: ["ml.t2.medium", "ml.m5.large"]
        transformInstanceTypes: ["ml.m5.large"]

    - name: CheckMSE
      type: Condition
      condition:
        conditions:
          - operator: LessThanOrEqualTo
            stepName: EvaluateModel
            propertyFile: EvaluationReport
            jsonPath: regression_metrics.mse.value
            threshold: 6.0
        ifSteps:
          - RegisterModel