snowplow-devops / terraform-aws-enrich-kinesis-ec2 Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
The current defaults are too aggressive for the Kinesis API limits and are forcing artificially scaling the Kinesis Stream to achieve moderate throughput.
Bump stream-enrich to 2.0.5
I recently set up the secure version of the AWS Quickstart terraform and upgraded all the Terraform modules to their latest version. The only issue I kept running is that the enricher seems to be calling its own $HOSTNAME
at some point (e.g. ip-10-10-3-119
or ip-10-10-3-119.us-west-2.compute.internal
) however that hostname is not available in /etc/hosts
so it throws and UnknownHostException. This is described in more detail on this Discourse thread.
I have solved this by forking this Terraform module and adding a custom_script
variable like so:
# main.tf
module "enrich_kinesis" {
source = "./enrich-kinesis-ec2-aws"
#version = "0.5.1"
...
custom_script = "echo \"127.0.0.1 $HOSTNAME\" >> /etc/hosts"
}
This is then referenced user-data.sh.tmpl
to make sure it runs when starting the EC2 instances:
# user-data.sh.tmpl
readonly CONFIG_DIR=/opt/snowplow/config
sudo mkdir -p $${CONFIG_DIR}
sudo cat << EOF > $${CONFIG_DIR}/enrich.hocon
${config}
EOF
sudo docker run \
...
${telemetry_script}
${custom_script}
This works for now, but I have a few considerations (and am happy to contribute where possible/needed):
One more observation, not sure if it's relevant is that I haven't been able to use the method in /snowplow-devops/terraform-aws-service-ec2/ to get the instance ID:
function get_instance_id() {
local __ec2_metadata_token
__ec2_metadata_token=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -H "X-aws-ec2-metadata-token: $${__ec2_metadata_token}" --silent --location "http://169.254.169.254/latest/meta-data/instance-id/"
}
However, I was able to use either $HOSTNAME
(for just the hostname) or the instance's local ec2-metadata utility /opt/aws/bin/ec2-metadata -i
to get the instance ID.
As reported here: https://discourse.snowplowanalytics.com/t/enrich-server-cannot-convert-configuration-enrichment-no-longer-validating-events/6846/6
If Telemetry is disabled then some strings are missing entirely which breaks the module.
Both the referer_parser and ua_parser_config are hardcoded to download assets from eu-west-1. This needs to be changed to HTTPS download links to avoid the regional download issue.
Required for QA modules
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.