snowplow-devops / terraform-aws-enrich-kinesis-ec2 Goto Github PK

The current defaults are too aggressive for the Kinesis API limits and are forcing artificially scaling the Kinesis Stream to achieve moderate throughput.

1. The defaults should minimise cost and maximise throughput
2. The buffers should be configurable so a user can redefine these limits for their own tests

Bump stream-enrich to 2.0.5

I recently set up the secure version of the AWS Quickstart terraform and upgraded all the Terraform modules to their latest version. The only issue I kept running is that the enricher seems to be calling its own $HOSTNAME at some point (e.g. ip-10-10-3-119 or ip-10-10-3-119.us-west-2.compute.internal) however that hostname is not available in /etc/hosts so it throws and UnknownHostException. This is described in more detail on this Discourse thread.

I have solved this by forking this Terraform module and adding a custom_script variable like so:

# main.tf

module "enrich_kinesis" {
  source  = "./enrich-kinesis-ec2-aws"
  #version = "0.5.1"

  ...

  custom_script = "echo \"127.0.0.1  $HOSTNAME\" >> /etc/hosts"
}

This is then referenced user-data.sh.tmpl to make sure it runs when starting the EC2 instances:

# user-data.sh.tmpl
readonly CONFIG_DIR=/opt/snowplow/config

sudo mkdir -p $${CONFIG_DIR}
sudo cat << EOF > $${CONFIG_DIR}/enrich.hocon
${config}
EOF

sudo docker run \
... 

${telemetry_script}

${custom_script}

This works for now, but I have a few considerations (and am happy to contribute where possible/needed):

• Is there an underlying reason why the instance hostname is referenced in this way?
• Is there an underlying reason why the instance hostname is not available to the Java runtime/docker container?
• Might it be generally convenient to be able to add a custom script to the startup script of all EC2 instances just like it's possible to turn telemetry on/off
• I've only run into this for the enricher, not for any other services using EC2

One more observation, not sure if it's relevant is that I haven't been able to use the method in /snowplow-devops/terraform-aws-service-ec2/ to get the instance ID:

function get_instance_id() {
  local __ec2_metadata_token
  __ec2_metadata_token=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
  curl -H "X-aws-ec2-metadata-token: $${__ec2_metadata_token}" --silent --location "http://169.254.169.254/latest/meta-data/instance-id/"
}

However, I was able to use either $HOSTNAME (for just the hostname) or the instance's local ec2-metadata utility /opt/aws/bin/ec2-metadata -i to get the instance ID.

https://github.com/snowplow-devops/terraform-aws-enrich-kinesis-ec2/blob/main/variables.tf#L288

As reported here: https://discourse.snowplowanalytics.com/t/enrich-server-cannot-convert-configuration-enrichment-no-longer-validating-events/6846/6

If Telemetry is disabled then some strings are missing entirely which breaks the module.

Both the referer_parser and ua_parser_config are hardcoded to download assets from eu-west-1. This needs to be changed to HTTPS download links to avoid the regional download issue.

Required for QA modules