TypeError - handler is not a function about datadog-lambda-js

[tianchu]

Sorry for the late response, we noticed that certain Node14.x Lambda functions deployed in us-east-1 were observing intermittent errors (“handler is not a function”) between 7:20PM October 13th EDT and 8:00AM October 14th EDT. We are actively investigating the root cause with the AWS Lambda team, and will share the results with the community as soon as possible. Please subscribe this issue or contact Datadog support ([email protected]) for the investigation results and additional technical details.

[RealityCtrl]

I opened a support case with AWS and the initial response says the internal team "are actively investigating the increased errors when using Node.JS 14 in the US-EAST-1 Region." so this may be an AWS issue.
I recreated the lambda with the same code and version of the datadog layer and haven't see any issues for last hour or so.

[lielran]

Ok. Found a workaround for now.
the cdk construct is changing the handlers by injecting /opt/nodejs/node_modules/datadog-lambda-js/handler.handler and replace it with DD_LAMBDA_HANDLER value (the original handler).

by changing the lambda handler to the local file path + function solve the issue (the value you have in DD_LAMBDA_HANDLER).

we don't see more error the error no more

[Czechh]

Support got back to us.

Furthermore,  Lambda team performed rollback to fix this issue but this may take some time to fully propagate. However, I am actively monitoring internal ticket and I will keep you updated on this case. 

[alexis-caylent]

If it helps, my team has experienced the very same issue in ca-central-1 as well

[tianchu]

Since the issue has been fixed, I'm going to close this thread. We are writing a postmortem next week, please contact [email protected] for a copy as you desire. Again, very sorry for everyone's loss and inconvenience, we are doing everything we can to prevent similar issues from happening again! Thanks for your understanding and help! 🙇

[tianchu]

@Kamitenshi have you tried rebuild your image using the latest of public.ecr.aws/lambda/nodejs:14 and redeploy your function?

[aterreno]

Amazing @lielran that really seems to work - I think it's the only thing I haven't tried, I was going to just rip off datadog complitely

[Czechh]

Same for us, we had Node 14.x and the layer 49. I switched to 64 just to make sure it wasn't the layer version, but it quickly became evident that the version didn't matter. Since there was no code change on our side before and after the errors started showing up, we believe it's infra related.

The layer is still there, and you can't modify it AFAIK so this must be AWS.

We opened a ticket with AWS. I think they're are doing some changes to Lambda and this might be related

You are receiving this message because we identified that your account created or updated or invoked Lambda functions on or after July 1, 2021.

AWS Lambda is extending the capability to track the current state of a function through its lifecycle to all functions [1]. With this change, you may need to update your CLI and/or SDK-based automation workflows around creating and updating functions by adding a check that the function became active before performing additional actions that operate on the function.

Previously, states have been used in two primary use-cases. Firstly, to move the blocking setup of VPC resources out of the path of function invocation. Secondly, to allow the Lambda service to optimize new or updated container images for container-image based functions, also before invocation. By moving this additional work out of the path of the invocation, customers see lower latency and better consistency in their function performance.
Infrastructure as code and deployment automation tools such as AWS CloudFormation, AWS Serverless Application Model (SAM), AWS Cloud Development Kit (CDK), Serverless Framework,Hashicorp Terraform, AWS Chalice and Cloud Custodian already support states.

If you are using these tools and are already on the latest Software Development Kit (SDK), you do not need to take any action, and can stop reading now. If you are using an earlier SDK version, please update to the latest one.

We are rolling out this change over multi-phase periods to allow you to update existing tooling for deploying and managing Lambda functions. You can delay this change for your functions until December 5, 2021. Starting December 6, 2021, the delay mechanism expires and all customers see the Lambda states lifecycle applied during function create or update. Read this blog post [2] to learn about this change, timelines for different phases and a reference example on how to check your function state.

[DylanGaffney]

Same for us, we had Node 14.x and the layer 49. I switched to 64 just to make sure it wasn't the layer version, but it quickly became evident that the version didn't matter. Since there was no code change on our side before and after the errors started showing up, we believe it's infra related.

The layer is still there, and you can't modify it AFAIK so this must be AWS.

We opened a ticket with AWS. I think they're are doing some changes to Lambda and this might be related

You are receiving this message because we identified that your account created or updated or invoked Lambda functions on or after July 1, 2021.

AWS Lambda is extending the capability to track the current state of a function through its lifecycle to all functions [1]. With this change, you may need to update your CLI and/or SDK-based automation workflows around creating and updating functions by adding a check that the function became active before performing additional actions that operate on the function.

Previously, states have been used in two primary use-cases. Firstly, to move the blocking setup of VPC resources out of the path of function invocation. Secondly, to allow the Lambda service to optimize new or updated container images for container-image based functions, also before invocation. By moving this additional work out of the path of the invocation, customers see lower latency and better consistency in their function performance.
Infrastructure as code and deployment automation tools such as AWS CloudFormation, AWS Serverless Application Model (SAM), AWS Cloud Development Kit (CDK), Serverless Framework,Hashicorp Terraform, AWS Chalice and Cloud Custodian already support states.

If you are using these tools and are already on the latest Software Development Kit (SDK), you do not need to take any action, and can stop reading now. If you are using an earlier SDK version, please update to the latest one.

We are rolling out this change over multi-phase periods to allow you to update existing tooling for deploying and managing Lambda functions. You can delay this change for your functions until December 5, 2021. Starting December 6, 2021, the delay mechanism expires and all customers see the Lambda states lifecycle applied during function create or update. Read this blog post [2] to learn about this change, timelines for different phases and a reference example on how to check your function state.

We're experiencing this too and can confirm the workaround that @lielran posted above worked for us.

So is the problem here that the DD layer is generating the DD_LAMBDA_HANDLER environment variable using the relative path for the layer instead of the literal path for the bundled lambda? Did AWS indicate that this is something they actually intend to fix, or is the update they made going to be the new method for bundling to which we need to adapt?

[gpoitch]

Same issue has been happening for us with a simple lambda for the last 12 hours - no datadog or layers.
Stumbled upon this searching the internet for similar cases.

[tianchu]

@alexis-caylent I don't think aws:states:opt-out is related. Are you currently experiencing the same error in ca-central-1?

[tianchu]

@Kamitenshi and for whoever using container images, try use an older version of the base image by AWS public.ecr.aws/lambda/nodejs:14.2021.10.07.14-x86_64. According to our tests, any version after this one still experience the issue.

[tianchu]

LATEST UPDATES [12PM 10/15/21 EDT]: We no longer observe issues in ca-central-1 or functions using public.ecr.aws/lambda/nodejs:14. No customer is expected to experience this issue at this point time. If that's not the case, please let us know.

[lielran]

Same as here, we are seeing the same problem in us-east-1 we are using node layer version 55 and extension layer 8 on node 14.x

[avnersch]

+1, exactly the same issue.
For us, it started on October 13th, 11PM UTC time.

nodeLayerVersion: 55,
extensionLayerVersion: 8,

[AmitFidler]

Same here, same versions - 55/8 - it happens for us when enableDatadogTracing flag is set to both true/false.

[aterreno]

@AmitFidler so did you solve it? removing the flag enableDatadogTracing?

[AmitFidler]

@aterreno unfortunately no, the errors occurs when the enableDatadogTracing is both on and off.
We are using the aws-cdk constructs and those are the two configurations we have - both of them produce the described error:

        const datadog = new Datadog(this, 'Datadog', {
            nodeLayerVersion: 55,
            extensionLayerVersion: 8,
            flushMetricsToLogs: false,
            addLayers: true,
            apiKey: 'key',
            enableDatadogTracing: true,
            injectLogContext: false,
        });

        const datadog = new Datadog(this, 'Datadog', {
            nodeLayerVersion: 55,
            extensionLayerVersion: 8,
            flushMetricsToLogs: false,
            addLayers: true,
            apiKey: 'key',
            enableDatadogTracing: false,
            injectLogContext: true,
        });

[aterreno]

@AmitFidler yep I've tried a bit of everything. I've removed all the Ddog env variables, I've removed the LambdaInsights layer but no luck. It seems like intermittent, which makes it even harder to troubleshoot.
I also tried the latest layer image but no luck, we were on the previous version.

export const datadogEnvVars = {
  DD_LAMBDA_HANDLER: 'bundle.handler',
  // DD_TRACE_ENABLED: 'true',
  // DD_FLUSH_TO_LOG: 'true',
  // DD_MERGE_XRAY_TRACES: 'true',
  DD_API_KEY: 'key',
}

export const defaultLayers = [
  `arn:aws:lambda:${region}:464622532012:layer:Datadog-Node14-x:64`, (was 62)
  `arn:aws:lambda:${region}:464622532012:layer:Datadog-Extension:11`, (was 10)
  // `arn:aws:lambda:${region}:580247275435:layer:LambdaInsightsExtension:14`,
]

export const defaultRuntime = 'nodejs14.x'

// This handler is present in the datadog layer
export const defaultHandler = '/opt/nodejs/node_modules/datadog-lambda-js/handler.handler'

export const defaultArchitecture = 'x86_64'

[lielran]

I think I found the root cause.

[aterreno]

@lielran strange I don't see that error unless I am looking at a wrong lambda

[lielran]

@aterreno I've tried to add the layer again and the verify btn works. so I guess the layer do exist

[aterreno]

@lielran if I run

aws lambda get-layer-version-by-arn --arn arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Node14-x:64

I get back

{
    "Content": {
        "Location": "https://prod-04-2014-layers.s3.us-east-1.amazonaws.com/snapshots/464622532012/Datadog-Node14-x-a4ce8b59-0927-4509-b569-f2a2532fceec?versionId=9MKy531q1.1nQzqZFEKHJMQsTN8GKTip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJGMEQCIF8fqN5M1JactAEr9mTIMgdPXD9LOPzPgInyjDjBGxSZAiBw%2BqUO3UlZiLDyg9%2FPzPmbX3yc3fknS34b8bhHywSNcir6AwhBEAAaDDc0OTY3ODkwMjgzOSIMzepTF1yDHlskmoRbKtcDQ%2F9hTeqe9mBi0Yr%2Fzpv702KjXD6pe7dqCzq%2FuxZvxRi2X0AK5anid9JVTkGZjRY6p1UTiG0Tz4VuoaaRHcQrze0g3cmkkLtgDk7%2Fn2XcClSVZ8s27UzA59xlaEqcAgWQIfoIg%2B43xTw6gVonjzrEWY0lZqBBV2U%2FKvcsaZXs9je64n7CY%2FE2wergezIn42AulIepc21Wy9hrALbYDe3k4L6xM0HPj6tw1DV6kEdFiCuegFeln80MkuPaLLl6fMTF1aKfKzo%2BJRWlolbugFRS5ks6rkN6LXVyZ8lwutnHo7V8Ym4c3GBWv3Hf8%2Fpbk0zD47srBwQqU%2B9czjMemDD9ddc7jIkRC0TFKmtrdgU%2BiogENREcCQq8LLYn5bPAAisTIHDWY1yeghymTNvKOzWsYZtdeXK%2BY4Ly6xok14XmHZR6cJhVcElIYTnfIf5%2BgBnQJDRNPJdTWOdlFlWk8p3ltkaHbwi7pK3zv8wiW%2Fk0LPdU9qtPaCHcI9ddLk3j2fDJpGwZosYQtmSV%2Bkjdej7fFJIARS2uD54CjhonkhUkCjhXFOSLvIdPvJfXZSa%2FrBp0STL0ZIGdsonqMja86r%2BD6sjHzrFJ02qCBNJvgfl1i9C9NN%2FVgSyzMIrVn4sGOqYBYM5NvJxoMmInwXfls%2B1CHUY9Fg071cMcrNRwtEY3wKSv7pVZ1iZwhAHwYqXjxzF8gOvjJl7xz6nWPTyWRCvCP3TzJKP%2BxKKbVm9eIIPIevJXPXNSqYKXuNAcjruHKr0h7RZqgj6gyOQAlonQ9ZX8F48rb7oMyD1Eabeg6Oo%2Bo9XHqAOWZE6uYMzcT5eHbYlHKSBZDIM%2F6aoiFJnQva0ihPdaQQO%2FDg%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211014T094715Z&X-Amz-SignedHeaders=host&X-Amz-Expires=600&X-Amz-Credential=ASIA25DCYHY32PPT5WUI%2F20211014%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=de2b0cb4029d0e735ad16faae495189d91eeebe693e3700f2ac3fe71564f3888",
        "CodeSha256": "yZDr3JEFhZ0lMZEg/3py7FUVXvJLmw4nnFZ180UiuZM=",
        "CodeSize": 1694044,
        "SigningProfileVersionArn": "arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc",
        "SigningJobArn": "arn:aws:signer:us-east-1:464622532012:/signing-jobs/cdda29d2-781f-4fd9-a997-7c4aefd85bc3"
    },
    "LayerArn": "arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Node14-x",
    "LayerVersionArn": "arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Node14-x:64",
    "Description": "Datadog Lambda Layer for Node",
    "CreatedDate": "2021-10-05T18:40:56.840+0000",
    "Version": 64,
    "CompatibleRuntimes": [
        "nodejs14.x"
    ]
}

which seems ok. so I think they corrupted/borked the layer somehow

[lielran]

I've tried to remove it then add it back. seem to be good for now. still monitor on this one

[aterreno]

Ok I'll do the same.

[aterreno]

@lielran nope it didn't work for us; removed and added, and then also removed from the ui from a specific lambda (some other lambda seem fine).

I'll delete the lambda now and see how it goes.

[lielran]

Same as here, open a ticket. it seems like aws issue on us-east-1.
it feels like the lambda layer is falling to init in some cases then in runtime we are missing dependencies

[aterreno]

@Czechh interesting - we do use pulumi for deployment/infra automation, I'll make sure we are on the latest version, but we keep up-to-date quite well

[matvi3nko]

@Czechh was it AWS Lambda team or Datadog Lambda team? Thanks.
We have same issue in all environments.

Support got back to us.

Furthermore,  Lambda team performed rollback to fix this issue but this may take some time to fully propagate. However, I am actively monitoring internal ticket and I will keep you updated on this case. 

[Czechh]

@matvi3nko AWS team

[sdrioux]

We had (have?) this same issue. Seems like it resolved itself however. Maybe AWS fixed it? Anyone still seeing issues?

[aterreno]

I am not seeing it and I am not sure if it's thanks to @lielran workaround or because AWS team rolledback as said here #237 (comment)

[ProficientBell]

Any updates on what happened here? Wondering if it's worth hosting this layer in my aws account or just doing away with it entirely and moving to another tracing platform

[tianchu]

@alexis-caylent that's very helpful and good to know! Our self-monitoring apps were deployed in many regions (but not ca-central-1) and we only observed that error in us-east-1.

[alexis-caylent]

Still monitoring executions to confirm, but it would seem that setting aws:states:opt-out as the description for the affected function(s) fixes the issue.

[aterreno]

So the issue is still lingering?
Because changing my handler to the default one (so not the datadog one) sorted out the situation for me.

Does it mean I am not using datadog though? I see logs but perhaps no metrics?

- export const defaultHandler = '/opt/nodejs/node_modules/datadog-lambda-js/handler.handler'
+ export const defaultHandler = 'bundle.handler'

was my change

[alexis-caylent]

Still monitoring executions to confirm, but it would seem that setting aws:states:opt-out as the description for the affected function(s) fixes the issue.

I take it back. After a while the functions started failing again, at this point I'm not sure if opting out of the lambda state tracking update has any effect whatsoever. Again, this is ca-central-1, not sure if the behavior would be the same in us-east-1.

[alexis-caylent]

Yes, a few of our lambdas are still throwing the same error in ca-central-1.

[tianchu]

@alexis-caylent any chance you are using container image from ECR?

[alexis-caylent]

So the issue is still lingering? Because changing my handler to the default one (so not the datadog one) sorted out the situation for me.

Does it mean I am not using datadog though? I see logs but perhaps no metrics?

- export const defaultHandler = '/opt/nodejs/node_modules/datadog-lambda-js/handler.handler'
+ export const defaultHandler = 'bundle.handler'

was my change

Indeed, my understanding is that if you do that you are effectively bypassing the datadog layer, which is responsible for sending some custom serverless metrics to datadog. Logs should remain unaffected, as they are picked up from Cloudwatch by the datadog forwarder.

[alexis-caylent]

@alexis-caylent any chance you are using container image from ECR?

Nope, we are not using container images

[tianchu]

#237 (comment) is a good workaround to stop the errors, but you would also lost datadog monitoring data, because the change bypasses the datadog handler/wrapper. In addition to the changes described in that comment, if you want to keep datadog monitoring, you can apply the datadog handler/wrapper in your function code:

const { datadog } = require("datadog-lambda-js");

module.exports.myHandler = datadog(myHandler, {
  // my function code
});

[aterreno]

@alexis-caylent makes sense - so this is 100% still happening - @alexis-caylent probably worth opening a ticket also with AWS if you haven't done that yet

@tianchu me neither no container images, just layers and deployed via pulumi as mentioned above.

@tianchu yes we can wrap in code the functions again 'old school' but then we'll also have to include that dependency (we don't as we shrink the code with webpack)

I think I am gonna hold on to this until is solved, then revert to use the datadog handler

[tianchu]

@aterreno you don't have to include the datadog-lambda-js as a dependency, as it's built in our layer. If needed, you can also mark them as external for webpack to know it's already there and no need to bundle it
https://docs.datadoghq.com/serverless/troubleshooting/serverless_tracing_and_webpack.

[aterreno]

Oh I see so you mean wrap but keep the layer, makes sense

…

On Thu, Oct 14, 2021 at 4:54 PM Tian Chu ***@***.***> wrote: @aterreno <https://github.com/aterreno> you don't have to include the datadog-lambda-js as a dependency, as it's built in our layer. If needed, you can also mark them as external for webpack to know it's already there and no need to bundle it https://docs.datadoghq.com/serverless/troubleshooting/serverless_tracing_and_webpack . — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#237 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASSCWWRY5SSA5EEUEM3YLUG34KPANCNFSM5F7DPTWQ> .

[regischow]

This is a response I got from AWS Support.

Thank you for your patience while I took time to research your case further. I discovered that we were experiencing a widespread issue with Nodejs.14x. and DataDog in our Lambda service environment. The 1st impact we have reports of began on 10/13 at 10:11PM (PDT). This widespread was reported as resolved on 10/14 at 2:11PM (PDT).

Please let us know if you experience anymore intermittent errors from your Lambda functions after 5:11PM (EST). If so, please attach the full execution logs at the time of error, and I will review it with our internal team for further investigation.

We do apologize for any inconvenience that this may have caused. If you have any additional questions or concerns, please do not hesitate to reach out. Thank you.

We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center.

[Kamitenshi]

Still experiencing this issue in eu-west-3
We are using container image

[aterreno]

It's a shame that you can't run a lambda on a specific version of that image (without using docker), that's clearly what's going on under the hood. so for that image this is the docker history: ``` IMAGE CREATED CREATED BY SIZE COMMENT sha256:457fc87038b157dfdf8e0ed54e95204c58de8b9959f098bacfd9159583a58c04 7 days ago ENTRYPOINT [ "/lambda-entrypoint.sh" ] 0B <missing> 7 days ago ENV LAMBDA_RUNTIME_DIR=/var/runtime 0B <missing> 7 days ago ENV LAMBDA_TASK_ROOT=/var/task 0B <missing> 7 days ago ENV LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib 0B <missing> 7 days ago ENV PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin 0B <missing> 7 days ago ENV TZ=:/etc/localtime 0B <missing> 7 days ago ENV LANG=en_US.UTF-8 0B <missing> 7 days ago WORKDIR /var/task 0B <missing> 7 days ago ADD file:354331510dc87abaf9c147399a5d89b33af0c8e7f604f38ebee7a06cde30fc80 / 64MB <missing> 7 days ago ADD file:d953b6b0983d5b100a7556347a418891ce5de00ac08e6e2da4d3b8590b0c5c62 / 104MB <missing> 7 days ago ADD file:a255bd5066e818e03b2981273d1021319564f22f1802f433259f39f6327858a5 / 8.2MB <missing> 7 days ago ADD file:12aa31059fba96439f8283ca085c03da7661aecb82c835878dcc56a000ae2d78 / 397B <missing> 7 days ago ADD file:00bf99ecf8ab8def901174a2668f1b54cb91a2f2153a9ca66fad414bdb5bbadf / 454kB <missing> 7 days ago ADD file:0a9e51a0b20dec22764b49cbb71ce89d6e7de8319c3bc66bd828e736af371fea / 295MB <missing> 7 days ago ARCHITECTURE amd64 0B ``` on public.ecr.aws/lambda/nodejs:14.2021.10.14.13-x86_64 which seems to be the latest ``` IMAGE CREATED CREATED BY SIZE COMMENT sha256:638af4ecee5724e33f02db2be0fd93a20936617598ff20edd1b5028231cc0e81 24 hours ago ENTRYPOINT [ "/lambda-entrypoint.sh" ] 0B <missing> 24 hours ago ENV LAMBDA_RUNTIME_DIR=/var/runtime 0B <missing> 24 hours ago ENV LAMBDA_TASK_ROOT=/var/task 0B <missing> 24 hours ago ENV LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib 0B <missing> 24 hours ago ENV PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin 0B <missing> 24 hours ago ENV TZ=:/etc/localtime 0B <missing> 24 hours ago ENV LANG=en_US.UTF-8 0B <missing> 24 hours ago WORKDIR /var/task 0B <missing> 24 hours ago ADD file:82cd367bba1dc4db6350271954c6ff94c583377921b339dc393f6466ac939c3b / 66.4MB <missing> 24 hours ago ADD file:99f0cc7a00f7b1263d9638d702be33afae185e7e2dad190bd8792d7d5b183af4 / 104MB <missing> 24 hours ago ADD file:a255bd5066e818e03b2981273d1021319564f22f1802f433259f39f6327858a5 / 8.2MB <missing> 24 hours ago ADD file:025d4367831d210e05ebf2fa6f1d2de930ff0e6d6e6643a57bb26fc68de39559 / 397B <missing> 24 hours ago ADD file:a362748d353dd59d5734f2f32f6e1e547c037d75e820d4c9e07d05e0ce65a931 / 454kB <missing> 24 hours ago ADD file:70c19ef3a1072aa6530754e3521c6e51ca69954fe91029b16135433f04df361c / 295MB <missing> 24 hours ago ARCHITECTURE amd64 0B ``` The 66.4MB layer is suspicious vs the old one 64MB, everything else seems more or less the same, also suspicious is that it was released 24 hours ago, i.e. when the problem started

…

On Fri, Oct 15, 2021 at 2:07 PM Tian Chu ***@***.***> wrote: @Kamitenshi <https://github.com/Kamitenshi> and for whoever using container images, try use an older version of the base image by AWS public.ecr.aws/lambda/nodejs:14.2021.10.07.14-x86_64. According to our tests, any version after this one still experience the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#237 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASSCQHM6VK3WNMXYZNN23UHARQZANCNFSM5F7DPTWQ> .

[tianchu]

@aterreno yes, we have asked AWS to see if they can republish the latest image with the content from the image published on 2021.10.07. I'm very sorry for the inconvenience!

[aterreno]

NP - not your fault - it's even more a shame that they don't publish the sources of these images, otherwise we could help fixing ;)

…

On Fri, Oct 15, 2021 at 2:34 PM Tian Chu ***@***.***> wrote: @aterreno <https://github.com/aterreno> yes, we have asked AWS to see if they can republish the latest image with the content from the image published on 2021.10.07. I'm very sorry for the inconvenience! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#237 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASSCTNKNU7XW66I7ATWJDUHAUXXANCNFSM5F7DPTWQ> .

[tianchu]

LATEST UPDATES [10AM 10/15/21 EDT]: We are still actively investigating with AWS on the root cause. It appears that the same issue is still affecting Lambda functions deployed in:

• ca-central-1 (according to customer reports)
• using container image public.ecr.aws/lambda/nodejs:14

Here are the workarounds:

For Lambda functions using container image public.ecr.aws/lambda/nodejs:14: Please use image public.ecr.aws/lambda/nodejs:14.2021.10.07.14-x86_64 instead, which should help fix the issue until the latest image gets updated again.

For Lambda functions in ca-central-1: Make the following changes to your functions until the issue resolves:

1. Set the handler function setting to your original handler function name (instead of the Datadog one), you can find your original handler value from the env var DD_LAMBDA_HANDLER.
2. Remove the env var DD_LAMBDA_HANDLER. With step 1 and 2, you would bypass the Datadog handler/wrapper, and stop the error. Follow the step below to keep receiving your telemetries (enhanced metrics, traces, etc.) in Datadog.
3. Apply the datadog wrapper and initialize the tracer in your function code

   const { datadog } = require("datadog-lambda-js");
   const tracer = require('dd-trace').init({});
   
   module.exports.myHandler = datadog(myHandler, {
     // my function code
   });

[Kamitenshi]

We are still experiencing this issue in eu-west-3 with the image public.ecr.aws/lambda/nodejs:14

[Kamitenshi]

@tianchu we do not have the issue anymore
The problem we had was due to Circle CI caching

[phil-gratifypay]

I still have this problem after I upgraded docker to 4.1.1.
The lambda function is in a NodeJS environment. The image is nodejs14.x-x86_64-9a2e4373acc332cd69e90c7f3.
This image might caused the problem because the function can successfully run in an earlier version. But, how can I specify a particular image for the lambda function if I want to use an earlier version of NodeJS environment?

[tianchu]

@phil-gratifypay you can see the usage tab of https://gallery.ecr.aws/lambda/nodejs, and the latest version public.ecr.aws/lambda/nodejs:14 should work. Alternatively, you can upgrade your datadog-lambda-js to the latest version https://github.com/DataDog/datadog-lambda-js/releases/tag/v4.65.0, which should help stop the error as well.

[phil-gratifypay]

@tianchu Thanks for your help. After I have changed the value of nodeLayerVersion to 65 and extensionLayerVersion to 13 in constructor of Datadog, the problem is gone.
new Datadog(this, "Datadog", { nodeLayerVersion: 65, extensionLayerVersion: 13, ... })

[tedyu]

@tianchu
What's the recommended version of datadog-lambda-js release for node.js 16 and 18 releases, respectively ?

Thanks