Extend support for cluster/namespace scoping of healthcheck about active-monitor

[RaviHari]

I would like to pick this up. Please let me know if this is okay.

[davemasselink]

@RaviHari, absolutely! the description above is fairly terse... so be sure to post here or reach out to me directly if any questions or opportunities for clarification come up.

Thanks!

[davemasselink]

hey @RaviHari, I had the chance to sync with @shrinandj this afternoon on the topic

the most important aspects coming out of the discussion are:

• this feature had NOT been supported in the previous instantiation of the project, therefore it will be wholly new
• "scope" may not be the best term to describe the feature, it's already overloaded in the k8s/crd world in this way: https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#create-a-customresourcedefinition
• it IS a feature which @shrinandj sees value in building

by default, healthchecks are effectively "cluster" scoped. the work will be to support an example which he and I ran thru on the whiteboard

Use Case for namespace "scoping":

Ensure that pods running in a user namespace (not health) are able to connect to an RDS instance or access certificates via AWS Certificate Manager (ACM) using a IAM role which only namespace entities would have access to.

Details of text in whiteboard image (with some modifications for clarity):

1+2. Run healthcheck using cluster level "scope"; ensure that the child argo workflow runs in the same namespace as its parent healthcheck
3. Introduce support for scope (or "domain" or "realm" or some better, less overloaded term) in healthcheck spec.
4. Re-run test in step 2 but WRT non-health namespace
5. Ensure that each healthcheck's status values accurately reflect metrics ONLY for child workflow in corresponding namespace.

[RaviHari]

Thanks @davemasselink and @shrinandj for the details