lambda functions need garbage collection if removed from policy file about cloud-custodian

[thisisshi]

@robertsigler I'm definitely in favor of including a --mugc flag in the cli to do automated garbage collection of lambda policies but the problem i'm still facing here is that again, custodian doesn't have a definitive list of all policies that are supposed to be running in an account. adding a tag for the name of the policy file seems like a start to fixing this.

The original proposal for changing the prefix wouldn't be something that custodian would do "under the hood". Users would need to explicitly specify the function-prefix inside of their policies.

[jantman]

It seems to me that tagging the lambda funcs could help this... if each separate system running vault tagged the lambdas with a unique tag, they should be able to clean up only their own resources...

[thomasbiddle]

This isn't a default feature? If I remove something from a configuration; I'd expect it to be removed from my infrastructure. Kind of ironic that Cloud Custodian doesn't cleanup after itself - this may be a non-starter for me.

[jantman]

@desponda There's a separate tool for this in the git repository, called mugc.py in the tools/ops/ directory, with documentation in tools/ops/README.md

[kapilt]

its available in multiple forms, mugc.py as well policylambda.py (same dir) which is a cfn style deploy tool.

its something we need to do additional engineering work on today i agree, but its more targeted atm for additional serverless provider implementations for parity

[kapilt]

so its not part of the custodian cli, but i've dropped a mugc.py script in tools that will look at policy file and remove non current lambdas based on prefix and a cwe source. i'd like to add it to custodian proper, but the operation mode is implicit at the moment based on prefix and sniffing the event source, i'd like it to work against the deployed config file vs the current one to do a proper diff, but that will have to wait.

[kapilt]

even with mugc to cleanup, its a separate step, it does seems like we need to have this as part of the default run semantics. we'll need to store the config to the output location to do a diff during run and auto-cleanup.

[kapilt]

unfortunately its not quite as simple as a config file diff, as in many cases we have multiple config files run from separate systems targeting the account. at the moment using tools/mugc.py and passing in all config files intended to run against the account provides for the desired outcome.

[thisisshi]

@thomasbiddle we distribute a script: tools/ops/mugc.py to diff your aws lambda inventory against your current set of policies, as of now it's unsafe to delete during policy execution as a single policy file isn't necessarily the entire set of policies for a specific account. if you only use a single policy file, running mugc.py is straightforward and not much additional overhead. Perhaps we can include an option in the run command to include cleanup via mugc, but doing it by default is definitely not tenable imo

[thomasbiddle]

@thisisshi Makes sense. A flag to the run command would definitely make sense IMO!

When you state "single policy file" do you mean just that - a single policy file? Or do you mean a single run? (Pointing custodian at a set of files)

[thisisshi]

a single policy file containing all of the policies you would ever execute against your aws account or a set of files that are always executed together. For instance, policies can be separated as compliance/hygiene, or perhaps by cadence, a weekly/daily execution. In the case of splitting by cadence, you wouldn't be executing both at the same time, so there's a risk for removing the policies from one of the files when the other is executing. But, if you aren't doing that (running a single policy file or running the same set of files each time), you should be able to cleanup without fear of removing lambdas that should be there. running mugc as a separate tool enables you to consolidate all the policies together to ensure you're garbage collecting policies correctly.

[thisisshi]

another way around this would be to use the function-prefix option in the various lambda modes and restricting mugc to only target lambdas with the prefix you specify in the policy

possible solution:

policies:
  - name: prefixed-lambda
    mode:
      type: periodic
      function-prefix: custodian-1
...

$ custodian run policy.yml -s output -v --mugc --mugc-prefix=custodian-1

this would deploy a lambda with the name: custodian-1-prefixed-lambda and the mugc portion would only target lambdas that have the prefix custodian-1

we could extend this such that all weekly policies have the prefix custodian-weekly and dailies with custodian-daily to ensure that policies don't step on each other

[robertsigler]

@thisisshi It sounds like cleanup is already possible, provided that you provide a unique prefix for every policy file, and then call mugc for each policy file with that unique prefix. Is this something that could ever become the default behavior? It seems like a few of us were surprised when we removed a policy from our policy file and didn't see the deletion reflected in our account.

I expect people would be unhappy if their Lambda functions changed names without them 'opting in', so maybe Lambda functions could be tagged with the name of the policy file which manages them instead. Then, mugc could find all of the Lambda functions with that tag, and delete functions which are not defined in the policy file anymore.

There's still a use case through which an obsolete Lambda function could "leak" and go undeleted (that is, if you create a policy file with a cloudtrail policy in it, run cloud custodian with that policy file, and then delete the file), but I suspect that there is no way around this, so long as cloud custodian has no 'central authority' for which policies are supposed to exist in an account at a given time.

[robertsigler]

@thisisshi

adding a tag for the name of the policy file seems like a start to fixing this.

I understand that this is just one of many possible solutions, but is this something that is in Capital One's backlog? If not, is it something that you would accept a pull request for?

[thisisshi]

@robertsigler definitely something we would like to see implemented, either by us or via pull request

[desponda]

@kapilt @thisisshi did this issue fall out of the radar? Was there any final recommendation?

[natanlao]

I see this issue is still open, but is placed in the aws "Done" pipeline. Has this functionality made its way into the codebase (outside of mugc.py)?

[nitrocode]

What if there was an enabled flag for each policy that was always set to true. If you set any of the policies to false then when custodian ran, it would automatically delete the policy. The end result would be a disabled policy in code but at least it would be removed from the infrastructure.

policies:
  - name: prefixed-lambda
    enabled: false