snowplow-devops / terraform-aws-s3-loader-kinesis-ec2 Goto Github PK

There have been a number of updates to the S3 Loader, including a new configuration format. This issue is to move from v1 to v2 of the S3 Loader.

Hey team,

Is there a way to use "s3.partitionFormat" options in the terraform module?
https://docs.snowplow.io/docs/pipeline-components-and-applications/loaders-storage-targets/s3-loader/configuration-reference/

Example

module "s3_loader_enriched" {
  source  = "snowplow-devops/s3-loader-kinesis-ec2/aws
  ... other configs ...
  
  s3_partition_format = "date={yy}-{mm}-{dd}"
}

The current defaults are too aggressive for the Kinesis API limits and are forcing artificially scaling the Kinesis Stream to achieve moderate throughput.

1. The defaults should minimise cost and maximise throughput
2. The buffers should be configurable so a user can redefine these limits for their own tests

Side note for this app is that the byte limit is especially important when scaling as it can cause OOM issues when multiple shards are assigned to a single worker.