Route53: dynamic record triggers update even with no changes to the record about octodns-route53

[warmfusion]

Looks like the issue is with regards to the healthcheck elements;

Longer log output

2020-08-05T20:11:50  [139819830553928] INFO  Route53Provider[route53] plan: desired=example.com.
2020-08-05T20:11:51  [139819830553928] INFO  Route53Provider[route53] populate:   found 53 records, exists=True
2020-08-05T20:11:51  [139819830553928] INFO  Route53Provider[route53] _extra_changes_update_needed: health-check caused update of www.example.com.:CNAME
2020-08-05T20:11:51  [139819830553928] INFO  Route53Provider[route53] _extra_changes_dynamic_needs_update: health-check caused update of www.example.com.:CNAME
2020-08-05T20:11:51  [139819830553928] INFO  Route53Provider[route53] plan:   extra changes
  Update
    <CnameRecord CNAME 300, www.example.com., a.web.example.com., {'na': {'fallback': 'eu', 'values': [{'value': 'elb-long-things-go-here.amazonaws.com.', 'weight': 1}]}, 'eu': {'fallback': None, 'values': [{'value': 'vif07.web.example.com.', 'weight': 1}, {'value': 'vif08.web.example.com.', 'weight': 1}]}}, [{'pool': 'eu', 'geos': ['AF', 'AS', 'EU']}, {'pool': 'na', 'geos': ['NA', 'OC', 'SA']}, {'pool': 'eu'}]> ->
    <CnameRecord CNAME 300, www.example.com., a.web.example.com., {'na': {'fallback': 'eu', 'values': [{'value': 'elb-long-things-go-here.amazonaws.com.', 'weight': 1}]}, 'eu': {'fallback': None, 'values': [{'value': 'vif07.web.example.com.', 'weight': 1}, {'value': 'vif08.web.example.com.', 'weight': 1}]}}, [{'pool': 'eu', 'geos': ['AF', 'AS', 'EU']}, {'pool': 'na', 'geos': ['NA', 'OC', 'SA']}, {'pool': 'eu'}]> (config)
2020-08-05T20:11:51  [139819830553928] INFO  Route53Provider[route53] plan:   Creates=0, Updates=1, Deletes=0, Existing Records=53

The yaml in this case includes;

# For example.com.yaml
www:
  dynamic:
    pools:
      na:
        fallback: eu
        values:
        - value: elb-long-things-go-here.amazonaws.com.
      eu:
        values:
        - value: vif07.web.example.com
        - value: vif08.web.example.com
    rules:
    - geos:
      - AF
      - EU
      - AS
      pool: eu
    - geos:
      - NA
      - SA
      - OC
      pool: na
    - pool: eu
  octodns:
    healthcheck:
      host: a-different-domain-for-reasons.com
      path: /healthz
      port: 80
      protocol: HTTP
    route53:
      healthcheck:
        measure_latency: false

I wonder if the issue lies in the healthcheck syncing element not discovering the healthchecks and linking them up, so tries to 'create' them each time?

Error message returned from this function; https://github.com/github/octodns/blob/v0.9.10/octodns/provider/route53.py#L1231-L1258

We are using a bundled instance of v0.9.10

[warmfusion]

Attached a debug to the stack and i think I've identified the issue we're having;

https://github.com/github/octodns/blob/v0.9.10/octodns/provider/route53.py#L1049-L1053

        return host == config['FullyQualifiedDomainName'] and \
            path == config['ResourcePath'] and protocol == config['Type'] \
            and port == config['Port'] and \
            measure_latency == config['MeasureLatency'] and \
            value == config_ip_address

The issue seems to be where the 'host' == 'fqdn' check runs;

in my specific example using real data

host = 'techradar.com'
config['FullyQualifiedDomainName'] = 'vif07.web.future.net.uk.'

Seems theres some sort of inconsistency between the healthcheck data being pulled from AWS and the expectation of what that data needs to hold...

I shall continue my investigation...

[warmfusion]

So keeping things simpler and simpler; I created a new domain, without setting anything for octodns.healthcheck.host and the check still fails resulting in constant resync attempts.

toby:
  dynamic:
    pools:
      na:
        fallback: eu
        values:
        - value:  elb-long-things-go-here.amazonaws.com.
      eu:
        values:
        - value: vif07.web.example.com
        - value: vif08.web.example.com
    rules:
    - geos:
      - AF
      - EU
      - AS
      pool: eu
    - geos:
      - NA
      - SA
      - OC
      pool: na
    - pool: eu
  octodns:
    healthcheck:
      path: /.sre/healthz
      port: 80
      protocol: HTTP
    route53:
      healthcheck:
        measure_latency: false
  ttl: 300
  type: CNAME
  value: fte.web.example.com

You'll get 3 invokations of the _health_check_equivalent function where host == config['fqdn'] results in

• test.example.com == elb-long-things-go-here.amazonaws.com.
• test.example.com == vif07.web.example.com
• test.example.com == vif08.web.example.com

All of which fail, so all of which trigger an 'update'.

---

Feels like some sort of incompatibility between having octodns healthchecks attached to a dynamic route configuration as theres an assumption that the healthcheck is for the FQDN of the record, but when using dynamic entries this should be a healthCheck against the pool values themselves.

[ross]

We don't make use of dynamic CNAME so that functionality is unfortunately lightly tested :-/. Health checking of CNAMEs is way different than A/AAAA so it doesn't surprise me that there are issues there. I'm pretty swamped atm so won't have a chance to dig into it for a bit.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.