kharmabum / insight-pandemic Goto Github PK

Depends on #6

• KSQL + Grafana Demo: https://github.com/confluentinc/ksql/tree/0.1.x/ksql-clickstream-demo#clickstream-analysis

• Push data into Kafka (python script, cli): https://github.com/InsightDataScience/data-engineering-ecosystem/wiki/Kafka-advanced

• Prior to deployment, new event schemas ought to be validated to ensure data types are supported and fields match conventions.
• Prior to deployment, new queries ought to be validated to ensure appropriate resources (e.g. tables/topics) are available.
• In the case of an invalid entry, helpful error messages ought to be provided.

• Configure test topic (or auto-creating topics)
• Push data into Kafka (python script, cli): https://github.com/InsightDataScience/data-engineering-ecosystem/wiki/Kafka-advanced

The distributed deployment option adds fault tolerance and the option to horizontally scale throughput.

See https://stackoverflow.com/a/31371862

Deadline: 6/6/19, 5:30 PM

• README
• Google Slides https://docs.google.com/presentation/d/1vkE3Ajv-h3oT1M7pjeYCO7HaXr5QjviyjIslqmHGm5I

Requirements

• Project Idea
• Tech Stack
• Data Source
• Engineering Challenge
• Business Value
• MVP
• Stretch Goals

This would also be a good opportunity to look into scaling and upgrade management.

A PHSS must often import data from a number of static sources including disease registries, census data, annual health surveys, and administrative data systems (e.g. hospital discharge data) that are useful for monitoring public health and arrive in a variety of bulk formats.

• Add Avro Scheme + key on facility_id
• Run Schema Registry

• https://www.terraform.io/docs/providers/aws/r/s3_bucket.html

Tracking and visualizing a geo-spatial component of events would be useful for monitoring geographical trends.

See #12

To meet minimum regulatory requirements, data stored and in-transit should be secured at all times via encryption, client authentication, and policy-based, fine-grained access control.

• Provision resources
• Deploy S3 bucket
• Deploy S3 endpoint
• Push zipped config to S3
• Config instances
  • ssh
  • Download/unzip
  • on zookeeper nodes: echo "'[1, 2, 3]'" > /var/lib/zookeeper/myid,
  • start services (zookeeper nodes first)
• test ingestion

Installing Kafka

• Running in production: https://docs.confluent.io/current/kafka/deployment.html#cp-production-recommendations
• Manual installation: https://docs.confluent.io/current/installation/installing_cp/zip-tar.html#prod-kafka-cli-install
• System Requirements: https://docs.confluent.io/current/installation/system-requirements.html#system-requirements

Separate Terraform components (vpc vs services) and development environments, add remote state tracking

Example:

stage
  └ vpc
  └ services
      └ frontend-app
      └ backend-app
          └ vars.tf
          └ outputs.tf
          └ main.tf
          └ .terragrunt
  └ data-storage
      └ mysql
      └ redis
prod
  └ vpc
  └ services
      └ frontend-app
      └ backend-app
  └ data-storage
      └ mysql
      └ redis
mgmt
  └ vpc
  └ services
      └ bastion-host
      └ jenkins
global
  └ iam
  └ route53

• https://docs.confluent.io/current/control-center/installation/install-apache-kafka.html
• https://docs.confluent.io/current/installation/installing_cp/zip-tar.html#prod-kafka-cli-install

• VPC
• Security group to allow ssh
• Security group to allow downloads (from everywhere)
• Security group to allow inter-node communication w/i group

Security Groups:

- Basic https://docs.google.com/document/d/1ErNGLnbQPqpwL-SxlRRixj7FNOXWhCu0cqfXTsDos88/edit#
- Advanced https://docs.google.com/document/d/1odIxUQadfMGh08eadxP31nVN1-NYVRbkZ_5u6v2mo3o/edit#

Terraform

• https://github.com/insightDataScience/aws-ops-insight
• https://github.com/gruntwork-io/intro-to-terraform
• https://github.com/airpair/ntiered-aws-docker-terraform-guide/blob/edit/terraform/public-subnet.tf

Notes

• Check "connect" on instance tab for details for ssh
• VPC Endpoints
  • https://aws.amazon.com/premiumsupport/knowledge-center/connect-s3-vpc-endpoint/
  • https://docs.aws.amazon.com/vpc/latest/userguide/vpce-gateway.html
• S3 vs EBS: https://www.cloudberrylab.com/resources/blog/amazon-s3-vs-ebs-vs-efs/
• AWS Budgets:
  • https://aws.amazon.com/aws-cost-management/aws-budgets/
  • https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/budgets-create.html
  • https://console.aws.amazon.com/billing/home?#/budgets/create
• AWS Cost Explorer
  • https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-what-is.html
  • https://console.aws.amazon.com/cost-reports/home?#/dashboard

• Add ELB and auto-scaling group
• Manage scale-in, termination policy

• Terraform + Lambda: https://www.terraform.io/docs/providers/aws/r/lambda_function.html

User-defined queries that naturally translate to critical incidents should be able to deliver targeted alerts to users of the system.

• Installing KSQL: https://docs.confluent.io/current/ksql/docs/installation/installing.html
• KSQL Quick Start: https://docs.confluent.io/current/ksql/docs/quickstart.html?_ga=2.98770138.599741510.1559767301-1612029994.1557362581