goto / entropy Goto Github PK

Describe the bug
There is a warning when running entropy with dash in the app name

data-entropy-app-b7b7665-qxxl8 app 2023/09/04 06:17:31 Failed to export to Prometheus: cannot register the collector: descriptor Desc{fqName: "data-entropy_process_gc_sys", help: "Bytes of memory in garbage collection metadatas", constLabels: {}, variableLabels: []} is invalid: "data-entropy_process_gc_sys" is not a valid metric name

To Reproduce
Steps to reproduce the behavior:

1. Use dash in the app name
2. Run entropy

Expected behavior
No warning should be thrown and name should be valid

Firehose has a new dlq job feature that fetch dlq data from a dlq storage and send to a sink. Entropy need to support the orchestration of such job. We could make it generic enough to run kubernetes job without firehose or dlq context.

Requirements

• Run as a kubernetes job
  • Input
    • A set of configs and resource config
    • A dlq job image
  • Telegraf running as a sidecar to send metrics
• Kubernetes job is immutable
• Kubernetes job and pods won't be deleted unless it is over TTL (~ 1 month)
• Actions
  • Start
    • Create jobs and pods if not exist
  • Stop
    • Stopping the pods (not removing the pods since the logs need to be there for more debugging)
• Retention on finished jobs (Complete or Failed)
  • configurable ttl (default 1 month)
• Output
  • Need to get the final-state of jobs
    • Option 1: We could perhaps put the resource in PENDING if the status is not completed or failed so entropy would poll until it is COMPLETE. Entropy will the status as COMPLETE if it is either completed or failed.
    • Option 2: Store output when Get Resource (ref easiest)

Orchestration

There are 2 possible approaches of orchestration, using kubernetes client and using helm.
With kubernetes client, entropy would need to create individual resource one by one, configmap, metadata, etc or just creating 1 job resource and inject the configs and everything there. With helm, we will need to create a chart that would be used by this module.

1. Kubernetes client

Pros

• Simpler if we only create 1 job resource and injecting the configs in the resource

Cons

• Won't support versioning

2. Helm

Pros

• Have versioning

Cons

• Need to uninstall helm if we stop the job to leave no traces in the server

Ideas

• We can pass container spec as a config
• Job would have only 2 containers, 1 for the main and 1 for the telegraf

Decision

• Use kubernetes client and inject the configmap there