Usage

Deployed Resources

Once deployed, you should see the following in your AWS account:

Naming convention: <org>-<env>-<domain>-<module>-<resource>

SSM parameters: /<org>/<domain>/<module>/<resource-type>/<resource-name>/<attribute>

Resource	Deployed Name / SSM Path	Config Reference
S3 Bucket	`<org>-dev-shared-datalake-raw` `/<org>/shared/datalake/bucket/raw/name`	`buckets.raw` in `datalake/datalake.yaml`
S3 Bucket	`<org>-dev-shared-datalake-transformed` `/<org>/shared/datalake/bucket/transformed/name`	`buckets.transformed` in `datalake/datalake.yaml`
Glue Crawler	`<org>-dev-dataops-example-crawler-crawler1` `/<org>/dataops/dataops-crawler/crawler/example-crawler/name`	`crawlers.crawler1` in `dataops/crawler.yaml`
Glue Database	`<org>-dev-dataops-example-project-sample-database` `/<org>/dataops/dataops-project/database/name`	`projectName` in `dataops/project.yaml`
Athena Workgroup	`<org>-dev-shared-athena` `/<org>/shared/athena/workgroup/name`	configured in `datalake/athena.yaml`
IAM Role	`<org>-dev-shared-roles-data-admin` `/<org>/shared/generated-role/data-admin/id`	`generateRoles.data-admin` in `roles.yaml`
IAM Role	`<org>-dev-shared-roles-data-user` `/<org>/shared/generated-role/data-user/id`	`generateRoles.data-user` in `roles.yaml`
IAM Role	`<org>-dev-shared-roles-glue-etl` `/<org>/shared/generated-role/glue-etl/id`	`generateRoles.glue-etl` in `roles.yaml`

Assume the <org>-dev-shared-roles-data-admin role (via AWS CLI sts assume-role or console role switching).
Upload data files to the raw data bucket.
- Use SSM Parameter Store to find exact bucket names: /<org>/shared/datalake/bucket/<bucket-name>/name
See DATASETS.md for sample data you can use for testing.

Open the AWS Glue Console and run the deployed Glue Crawler to catalog uploaded data.
After the crawler completes, tables will appear in the Glue Data Catalog.