tiki-archive / platform-agg-lake

Deploy a data lake for use with mytiki.com

Aggregation: Lake

Deploy an encrypted Iceberg data lake to any AWS S3 account with a provisioned access point for management by mytiki.com.

How to Use

We're in private beta. You can apply for early discounted access and participation in our 90-day launch cohort.

Contributing

Please review our Contributor Guidelines before beginning. Utilize GitHub Issues, Discussions, and Security Advisories to provide feedback.

• Ask a Question
• Report a Bug
• Report a Vulnerability
• Request a Feature

platform-agg-lake's Issues

Workflow to publish publicly to SAR

User Story

#3

Description

1. Follow the template in: https://github.com/tiki-bar/sar-template
2. Update the policy to follow: https://docs.aws.amazon.com/serverlessrepo/latest/devguide/security_iam_resource-based-policy-examples.html#access-control-resource-based-example-share-publicly

Acceptance Criteria

• Application listed publicly in SAR

Create a SAR app for deploying a new data lake

Reason

Customers will need to be able to deploy a lake compatible with our software

Description

Using SAM and CloudFormation create a stack which when deployed in an AWS account creates a lake compatible with TIKI.

Include:

• S3 Bucket
• S3 Bucket Key (Use the AWS managed key)
• S3 Access Point with policy permissions for our account to read/write
• S3 Bucket Logging (Let the customer specify the logging bucket)

Deploy publicly to AWS SAR

References:

• https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html
• https://aws.amazon.com/serverless/serverlessrepo/
• https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html

Success Criteria

• template to deploy a lake
• gh action to deploy to sar
• able to register the lake with aws glue in a third-party account

Additional Information

No response

Lock down read/write lake access

User Story

#3

Description

Reading and writing to the lake from within the account should be limited to a single role.

Acceptance Criteria

• SAR app deploys an assumable role for access to the lake
• SAR app sets a bucket policy to limit access to the role

As a Customer, I want to automatically create Iceberg metadata on writes, so that I can query my staging tables

Acceptance Criteria

• On write to a data lake an event is sent to the metadata queue

Tasks

• TBD

Relates to

tiki-archive/platform-agg-metadata#107

Add deployment documentation

User Story

#3

Description

Add a README.md describing how to deploy and what resources to be created

Acceptance Criteria

• A description of resources created/manged
• An easy path deployment guide
• Documentation on all configuration options

As a Customer, I want to automatically create Iceberg metadata on writes, so that I can query my staging tables feature story

Acceptance Criteria

• On creation of a data lake an event is sent to the registration queue

Tasks

• TBD

Relates to

No response

As customer, I want to deploy a data lake, so that I can use mytiki.com

Acceptance Criteria

• A single deployable data lake
• Deployable to any aws account
• S3 based storage
• Iceberg compatability
• Support a centralized metadata catalog in Tiki accounts

Tasks

• #1
• #4
• #6
• #5

Relates to

No response