Lambda environment paramaters can exceed 4KB limit about terraform-aws-github-runner

[rsavage-nozominetworks]

Apparently changes to the runner labels have been made which seems to add additional labels to the respectful lambda environment variables. Due to this, it exceeded AWS 4kb limit and throws the error. We don't have many additional labels and they are fairly short. I can't image we are the only ones getting this error now.

I have since reverted back to 5.2.1 and the problem is gone.

[npalm]

Thanks for reporting. But 5.3.0 should not be breaking. It adds a feature for the multi runners to set a priority to the matcher. See here teh diff. Only terraform code 1829721

Do you know which parameter is causing the issue? Are you using a multi runner setup?

[npalm]

Adding ar ref to an AWS post, total size hsould be less than 4K, https://repost.aws/knowledge-center/lambda-environment-variable-size

Which mean we have to re-thing how we configure the lambda's

[npalm]

Can you share your configuration, of course without the sensitive part? So we can reproduce the isse.

[npalm]

Quickly had a chat with Amazon, there seems no easy way to pro-active warn on the size.

[npalm]

This script should give some insights in the sizes of the environment variables per function.

#!/bin/bash

functions=$(aws lambda list-functions --query 'Functions[*].FunctionName' --output text)

for function in $functions
do
  env_vars=$(aws lambda get-function-configuration --function-name $function --query 'Environment.Variables' --output text)
  size=$(echo -n $env_vars | wc -c)
  printf "%-5s bytes -> %s\n" $size $function
done

[rsavage-nozominetworks]

Thanks for reporting. But 5.3.0 should not be breaking. It adds a feature for the multi runners to set a priority to the matcher. See here teh diff. Only terraform code 1829721

Do you know which parameter is causing the issue? Are you using a multi runner setup?

I don't know which parameter is causing the issue persay. I just know that when I reverted back to 5.2.1, it stopped erroring with the 4KB lambda update limit. Yes, I am using the multi-runner setup.

[rsavage-nozominetworks]

Can you share your configuration, of course without the sensitive part? So we can reproduce the isse.

Yes, I can shared it once I clean up the sensitive details. Let me ask, what specifically are you looking for in regards to the configuration, the whole multi-runner setup? Are you wanting the module output or terraform code?

[rsavage-nozominetworks]

@npalm please let me know specifically what you are looking for and I'll scrub it and share it. Obviously, due to the possibility of sensitive information, I want to share at least as possible. Thanks!

[npalm]

I have checked our example, they are all on the save side of the 4KB limit. Also our productions deployments are all on the safe side. But indeed depending on how large names you chose and how many paramaters are defined the list can differ a lot. The potential problem exists on all labmda's. In my opnion the risk is the biggest on the webhook in combinaiton with the multi-runner. In that case each matcher (runner) creates on or more rules. Which ar passed as in a varaible.

The new release introduce an extra attribute. Which seems in your case the one that is exceeding the limit. Would like just to understand how big your setup is roughtly.

In the end we should find a solution. Which will be most likely moving away from the environment varialbes for Lambda.

I ran another experiament to calculate the environment variables. Which is using the module output for the webhook as josn. Look up the environment variables and pipe it to wc -c. This results seems to give a more accurate count.

[rsavage-nozominetworks]

These are the env vars from one of my runner lambdas:

ENVIRONMENT     :       vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
LOG_LEVEL       :       debug
MINIMUM_RUNNING_TIME_IN_MINUTES :       5
NODE_TLS_REJECT_UNAUTHORIZED    :       1
PARAMETER_GITHUB_APP_ID_NAME    :       /github-action-runners/prod-runners/app/github_app_id
PARAMETER_GITHUB_APP_KEY_BASE64_NAME    :       /github-action-runners/prod-runners/app/github_app_key_base64
POWERTOOLS_LOGGER_LOG_EVENT     :       true
RUNNER_BOOT_TIME_IN_MINUTES     :       5
SCALE_DOWN_CONFIG       :       [{\"cron\":\"* * 9-23 * * 1-5\",\"evictionStrategy\":\"oldest_first\",\"idleCount\":10,\"timeZone\":\"Europe/Amsterdam\"}]
SERVICE_NAME    :       runners-scale-down

And I have 11 runner types.

[npalm]

These are the env vars from one of my runner lambdas:

ENVIRONMENT     :       vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
LOG_LEVEL       :       debug
MINIMUM_RUNNING_TIME_IN_MINUTES :       5
NODE_TLS_REJECT_UNAUTHORIZED    :       1
PARAMETER_GITHUB_APP_ID_NAME    :       /github-action-runners/prod-runners/app/github_app_id
PARAMETER_GITHUB_APP_KEY_BASE64_NAME    :       /github-action-runners/prod-runners/app/github_app_key_base64
POWERTOOLS_LOGGER_LOG_EVENT     :       true
RUNNER_BOOT_TIME_IN_MINUTES     :       5
SCALE_DOWN_CONFIG       :       [{\"cron\":\"* * 9-23 * * 1-5\",\"evictionStrategy\":\"oldest_first\",\"idleCount\":10,\"timeZone\":\"Europe/Amsterdam\"}]
SERVICE_NAME    :       runners-scale-down

And I have 11 runner types.

This would not exceed 4KB, or does it?

[rsavage-nozominetworks]

These are the env vars from one of my runner lambdas:

ENVIRONMENT     :       vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
LOG_LEVEL       :       debug
MINIMUM_RUNNING_TIME_IN_MINUTES :       5
NODE_TLS_REJECT_UNAUTHORIZED    :       1
PARAMETER_GITHUB_APP_ID_NAME    :       /github-action-runners/prod-runners/app/github_app_id
PARAMETER_GITHUB_APP_KEY_BASE64_NAME    :       /github-action-runners/prod-runners/app/github_app_key_base64
POWERTOOLS_LOGGER_LOG_EVENT     :       true
RUNNER_BOOT_TIME_IN_MINUTES     :       5
SCALE_DOWN_CONFIG       :       [{\"cron\":\"* * 9-23 * * 1-5\",\"evictionStrategy\":\"oldest_first\",\"idleCount\":10,\"timeZone\":\"Europe/Amsterdam\"}]
SERVICE_NAME    :       runners-scale-down

And I have 11 runner types.

This would not exceed 4KB, or does it?

Nope, however using your script, it does show that the webhook lambda to be the culprit

3646  bytes -> nnnn-runners-webhook

[rsavage-nozominetworks]

Hi @npalm here is the output from the trying to go from 5.2.1 to 5.4.0

│ Error: updating Lambda Function (nnnn-runners-webhook) configuration: operation error Lambda: UpdateFunctionConfiguration, https response error StatusCode: 400, RequestID: c9a4f8f7-8320-4e57-8082-3ea8aadea5c3, InvalidParameterValueException: Lambda was unable to configure your environment variables because the environment variables you have provided exceeded the 4KB limit. String measured: 

{
    "POWERTOOLS_LOGGER_LOG_EVENT": "true",
    "POWERTOOLS_TRACER_CAPTURE_ERROR": "false",
    "RUNNER_CONFIG": '[{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-arm-2xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-arm-2xlarge-queued-builds.fifo","key":"linux-arm-2xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","arm64","nws-arm64-self-hosted-2xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-arm-4xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-arm-4xlarge-queued-builds.fifo","key":"linux-arm-4xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","arm64","nws-arm64-self-hosted-4xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-arm-large-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-arm-large-queued-builds.fifo","key":"linux-arm-large","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","arm64","nws-arm64-self-hosted"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-arm-xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-arm-xlarge-queued-builds.fifo","key":"linux-arm-xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","arm64","nws-arm64-self-hosted-xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-x64-2xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-x64-2xlarge-queued-builds.fifo","key":"linux-x64-2xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","x64","nws-self-hosted-2xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-x64-4xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-x64-4xlarge-queued-builds.fifo","key":"linux-x64-4xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","x64","nws-self-hosted-4xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-x64-large-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-x64-large-queued-builds.fifo","key":"linux-x64-large","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","x64","nws-self-hosted"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnn-x64-xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnn-x64-xlarge-queued-builds.fifo","key":"linux-x64-xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","linux","x64","nws-self-hosted-xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnnnn-x64-2xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnnnn-x64-2xlarge-queued-builds.fifo","key":"windows-x64-2xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","windows","x64","nws-windows-2xlarge"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnnnn-x64-large-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnnnn-x64-large-queued-builds.fifo","key":"windows-x64-large","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","windows","x64","nws-windows-large"]],"priority":999}},{"arn":"arn:aws:sqs:us-east-1:123456789012:nnnn-runners-nnnnnnn-x64-xlarge-queued-builds.fifo","fifo":true,"id":"https://sqs.us-east-1.amazonaws.com/123456789012/nnnn-runners-nnnnnnn-x64-xlarge-queued-builds.fifo","key":"windows-x64-xlarge","matcherConfig":{"exactMatch":true,"labelMatchers":[["self-hosted","windows","x64","nws-windows-xlarge"]],"priority":999}}]',
    "ENVIRONMENT": "nnnn-runners",
    "PARAMETER_GITHUB_APP_WEBHOOK_SECRET": "/github-action-runners/nnnn-runners/app/github_app_webhook_secret",
    "POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS": "false",
    "SQS_WORKFLOW_JOB_QUEUE": "",
    "LOG_LEVEL": "debug",
    "REPOSITORY_WHITE_LIST": "[]",
    "POWERTOOLS_TRACE_ENABLED": "false",
}

│
│   with module.multi_runners["nnnn"].module.multi_runner.module.webhook.aws_lambda_function.webhook,
│   on .terraform/modules/multi_runners.multi_runner/modules/webhook/webhook.tf line 9, in resource "aws_lambda_function" "webhook":
│    9: resource "aws_lambda_function" "webhook" {
│
╵

Looks like it's sitting at 4.37 KB. Something with the update added ~ .40 KB to the payload.

[npalm]

I think this requires we move the RUNNER_CONFIG to SSM.

[rsavage-nozominetworks]

Hi @npalm. I saw the recent actviity, and tried with the latest 5.4.2 version, however still getting the 4KB error

╷
│ Error: updating Lambda Function (xxxx-xxxxxxx-webhook) configuration: operation error Lambda: UpdateFunctionConfiguration, https response error StatusCode: 400, RequestID: 825e0324-8a73-4915-b24c-a7d6d05b18fd, InvalidParameterValueException: Lambda was unable to configure your environment variables because the environment variables you have provided exceeded the 4KB limit. String measured: 

{"POWERTOOLS_LOGGER_LOG_EVENT":"true","POWERTOOLS_TRACER_CAPTURE_ERROR":"false","RUNNER_CONFIG":"[{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-arm-2xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-arm-2xlarge-queued-builds.fifo\",\"key\":\"linux-arm-2xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"arm64\",\"xxx-xxxxx-self-hosted-2xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-arm-4xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-arm-4xlarge-queued-builds.fifo\",\"key\":\"linux-arm-4xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"arm64\",\"xxx-xxxxx-self-hosted-4xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-arm-large-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-arm-large-queued-builds.fifo\",\"key\":\"linux-arm-large\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"arm64\",\"xxx-xxxxx-self-hosted\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-arm-xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-arm-xlarge-queued-builds.fifo\",\"key\":\"linux-arm-xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"arm64\",\"xxx-xxxxx-self-hosted-xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-x64-2xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-x64-2xlarge-queued-builds.fifo\",\"key\":\"linux-x64-2xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"x64\",\"xxx-self-hosted-2xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-x64-4xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-x64-4xlarge-queued-builds.fifo\",\"key\":\"linux-x64-4xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"x64\",\"xxx-self-hosted-4xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-x64-large-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-x64-large-queued-builds.fifo\",\"key\":\"linux-x64-large\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"x64\",\"xxx-self-hosted\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-linux-x64-xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-linux-x64-xlarge-queued-builds.fifo\",\"key\":\"linux-x64-xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"linux\",\"x64\",\"xxx-self-hosted-xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-windows-x64-2xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-windows-x64-2xlarge-queued-builds.fifo\",\"key\":\"windows-x64-2xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"windows\",\"x64\",\"xxx-xxxxxxx-2xlarge\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-windows-x64-large-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-windows-x64-large-queued-builds.fifo\",\"key\":\"windows-x64-large\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"windows\",\"x64\",\"xxx-xxxxxxx-large\"]],\"priority\":999}},{\"arn\":\"arn:aws:sqs:us-east-1:012345678901:xxxx-xxxxxxx-windows-x64-xlarge-queued-builds.fifo\",\"fifo\":true,\"id\":\"https://sqs.us-east-1.amazonaws.com/012345678901/xxxx-xxxxxxx-windows-x64-xlarge-queued-builds.fifo\",\"key\":\"windows-x64-xlarge\",\"matcherConfig\":{\"exactMatch\":true,\"labelMatchers\":[[\"self-hosted\",\"windows\",\"x64\",\"xxx-xxxxxxx-xlarge\"]],\"priority\":999}}]","PARAMETER_GITHUB_APP_WEBHOOK_SECRET":"/github-action-runners/xxxx-xxxxxxx/app/github_app_webhook_secret","POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS":"false","SQS_WORKFLOW_JOB_QUEUE":"","LOG_LEVEL":"debug","REPOSITORY_WHITE_LIST":"[]","POWERTOOLS_TRACE_ENABLED":"false"}
│
│   with module.multi_runners["prod"].module.multi_runner.module.webhook.aws_lambda_function.webhook,
│   on .terraform/modules/multi_runners.multi_runner/modules/webhook/webhook.tf line 9, in resource "aws_lambda_function" "webhook":
│    9: resource "aws_lambda_function" "webhook" {
│
╵

[npalm]

The issue is not resolved yet. but I have refacored the lambda so the next step to move the paramters to SSM is much simpler. Help / PR's are welcome

[rsavage-nozominetworks]

The issue is not resolved yet. but I have refacored the lambda so the next step to move the paramters to SSM is much simpler. Help / PR's are welcome

Thanks. Unfortunately I don't have the experience or time to devote to refactoring this code base.

[npalm]

No problem, we keep the issue open. Hopefully our community has some time to provide a fix.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.

[rsavage-nozominetworks]

Hey guys, any way to get this prioritized? It's blocking us from updating at the moment, as we're falling way behind.

[npalm]

It is on our radar, but would love to get some community support here.

[spcaipers-arm]

Hi @npalm I have a local improvement for that. I am moving RUNNER_CONFIG to SSM as you suggested.
Question: Do you expect the ENV VAR name RUNNER_CONFIG on the webhook to change?

I changed its value to Parameters Store's name.
I am not sure what your lambda functions expect.

[spcaipers-arm]

I just saw there is a PR for that, so I will for that to be merged.

[rsavage-nozominetworks]

Thank you guys!