AWS infrastructure for dbt-athena automated and integration testing.
This is currently deployed on an AWS Account which is under the direct responsibility of @mattiamatrix.
AWS infrastructure for dbt-athena automated and integration testing
AWS infrastructure for dbt-athena automated and integration testing.
This is currently deployed on an AWS Account which is under the direct responsibility of @mattiamatrix.
This issue lays out a more restrictive permissions setup for Athena that does not allow direct access to S3. I've created an IAM Role & Policy that captures what I think those permissions are. Besides documentation, running a applicable subset of the functional tests against this role should give much greater confidence that the functionality works in the real world.
Resources:
DbtRestrictiveRole:
Type: AWS::IAM::Role
Properties:
RoleName: DbtAthenaRestricted
AssumeRolePolicyDocument: {
"Version" : "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Principal": {
"AWS": !Join [ ":", ["arn", "aws", "iam", "", !Ref "AWS::AccountId", "root"]]
},
"Action": [ "sts:AssumeRole" ]
} ]
}
Description: Minimal permissions policy to run dbt-athena
Policies:
- PolicyName: DbtRestrictive
PolicyDocument: {
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"athena:ListDataCatalogs",
"athena:GetDataCatalog",
"athena:ListDatabases",
"athena:GetDatabase",
"athena:StartQueryExecution",
"athena:StopQueryExecution",
"athena:GetQueryExecution",
"athena:GetQueryResults",
"athena:GetWorkGroup",
"athena:BatchGetQueryExecution"
],
"Effect": "Allow",
"Resource": "*",
"Sid": "AthenaPermissions"
},
{
"Action": [
"glue:CreateTable",
"glue:UpdateTable",
"glue:DeleteTable",
"glue:GetDatabases",
"glue:GetDatabase",
"glue:GetTable",
"glue:GetTables"
],
"Effect": "Allow",
"Resource": "*",
"Sid": "GlueAthenaPermissions"
},
{
"Action": [
"glue:CreateDatabase",
"glue:DeleteDatabase"
],
"Effect": "Allow",
"Resource": "*",
"Sid": "FuncTestPermissions"
},
{
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": "*",
"Sid": "S3ViaAthenaPermissions",
"Condition": {
"ForAnyValue:StringEquals": {
"aws:CalledVia": "athena.amazonaws.com"
}
}
}
]
}
Add a spark work group in the same catalog as the current work group that executes functional tests.
Preferred name of the work group: spark
Preferred engine config: lowest possible
Preferred session timeout: 1 hour.
Note: Do not create them in multiple catalogs since Athena spark does not have multi catalog support yet.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.