thekevjames / ifttt Goto Github PK

This might be a bit complex, but it'd be awesome to be able to do the following:

- name: update GKE max-autoscale-size
  watch:
    source: gke
    cluster: my-cluster
    pool: my-pool
  if: num_instances > 90% of max_instances
  then:
    - update-max-instances
    - send-slack-notif -t "my-pool was near its maximum, bumped up the size"

To reduce some complexity in action scripts, it'd be great if we could interpolate math. For example:

- name: ensure each user has seven nodes
  watch: ...
  then:
    - set-node-count {value * 7}

This issue was generated by todo based on a TODO: comment in 1bcf6b9. It's been assigned to @TheKevJames because they committed the code.

It'd be great if we could set time windows for certain tasks. Use cases: maintenance windows, alerting during office hours, etc. For example:

- name: update version of x
  watch:
    source: datastore
    kind: UpstreamVersions
    field: version_number
  then:
    - update-version --project {id} --version {value}
    - send-slack-notif -t "Updated {id} to {value}"
  between: 1AM and 3AM

This might be a good use-case for an optional if subkey? ie.

if:
  fn: curr > prev
  time: 1AM - 3AM

If would be great to be able to do:

- name: enable thing
  if: value
  ...

- name: change thing
  after: "enable thing"

In the case where change thing can only be run after enable thing completes.

This issue was generated by todo based on a TODO: comment in 1bcf6b9. It's been assigned to @TheKevJames because they committed the code.

Use case:

- name: alert on unprocessed jobs
  watch:
    source: elasticsearch
    connection_string: ...
    index: es-index
    query: {"filtered": {"filter": {"range": {"created": {"gte": "now-10m"}}}}}
  if: !curr['hits']['total']
  then:
    - send-slack-notif -t "no jobs processed in last ten minutes!"

This issue was generated by todo based on a TODO: comment in 1bcf6b9. It's been assigned to @TheKevJames because they committed the code.

As it stands, we can trigger and event, fail that event, and never attempt to re-process it. There should be some "dirty" bit set which denotes a failed action and can (optionally?) attempt to retry. Another option is to rollback the cache database in case of detectable processing errors, so that they are re-attempted on the next poll interval.

Some thoughts:

• Retry loops: eventually, the rollback or dirty-bit solution would need to de dead-lettered. Potential solution: IFTTT-FailedTasks could store attempts and their counts: on failure increment, then rollback if the count is below some threshold. If above some threshold, do not rollback.
• Rollback: we should probably avoid committing until after a success. Need to investigate race conditions here (double-processing is bad!!)
• Alerting: another option is to send an alert on failure. Maybe an on_fail key? Maybe v1 of this issue is to do this after a single failure with no rollback/retry/etc, then alert after n fails later?