dbt-athena / dbt-athena-infra

AWS infrastructure for dbt-athena automated and integration testing

dbt-athena-infra

AWS infrastructure for dbt-athena automated and integration testing.

This is currently deployed on an AWS Account which is under the direct responsibility of @mattiamatrix.

dbt-athena-infra's Issues

No role to ensure dbt-athena works against restrictive permissions

Re: dbt-athena/dbt-athena#302

This issue lays out a more restrictive permissions setup for Athena that does not allow direct access to S3. I've created an IAM Role & Policy that captures what I think those permissions are. Besides documentation, running a applicable subset of the functional tests against this role should give much greater confidence that the functionality works in the real world.

Resources:
  DbtRestrictiveRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: DbtAthenaRestricted
      AssumeRolePolicyDocument: {
          "Version" : "2012-10-17",
          "Statement": [ {
            "Effect": "Allow",
            "Principal": {
                "AWS": !Join [ ":", ["arn", "aws", "iam", "", !Ref "AWS::AccountId", "root"]]
            },
            "Action": [ "sts:AssumeRole" ]
          } ]
      }
      Description: Minimal permissions policy to run dbt-athena
      Policies:
        - PolicyName: DbtRestrictive
          PolicyDocument: {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": [
                        "athena:ListDataCatalogs",
                        "athena:GetDataCatalog",
                        "athena:ListDatabases",
                        "athena:GetDatabase",
                        "athena:StartQueryExecution",
                        "athena:StopQueryExecution",
                        "athena:GetQueryExecution",
                        "athena:GetQueryResults",
                        "athena:GetWorkGroup",
                        "athena:BatchGetQueryExecution"
                    ],
                    "Effect": "Allow",
                    "Resource": "*",
                    "Sid": "AthenaPermissions"
                },
                {
                    "Action": [
                        "glue:CreateTable",
                        "glue:UpdateTable",
                        "glue:DeleteTable",
                        "glue:GetDatabases",
                        "glue:GetDatabase",
                        "glue:GetTable",
                        "glue:GetTables"
                    ],
                    "Effect": "Allow",
                    "Resource": "*",
                    "Sid": "GlueAthenaPermissions"
                },
                {
                    "Action": [
                        "glue:CreateDatabase",
                        "glue:DeleteDatabase"
                    ],
                    "Effect": "Allow",
                    "Resource": "*",
                    "Sid": "FuncTestPermissions"
                },
                {
                    "Action": [
                        "s3:ListBucket",
                        "s3:GetBucketLocation",
                        "s3:GetObject",
                        "s3:ListBucketMultipartUploads",
                        "s3:ListMultipartUploadParts",
                        "s3:AbortMultipartUpload",
                        "s3:PutObject",
                        "s3:DeleteObject"
                    ],
                    "Effect": "Allow",
                    "Resource": "*",
                    "Sid": "S3ViaAthenaPermissions",
                    "Condition": {
                        "ForAnyValue:StringEquals": {
                            "aws:CalledVia": "athena.amazonaws.com"
                        }
                    }
                }
            ]
        }

Add spark work group

Description

Add a spark work group in the same catalog as the current work group that executes functional tests.

Additional information

Preferred name of the work group: spark
Preferred engine config: lowest possible
Preferred session timeout: 1 hour.

Note: Do not create them in multiple catalogs since Athena spark does not have multi catalog support yet.