datadog / datadog-lambda-js

The Datadog AWS Lambda Library for Node

datadog-lambda-js

Datadog Lambda Library for Node.js (16.x, 18.x, and 20.x) enables enhanced Lambda metrics, distributed tracing, and custom metric submission from AWS Lambda functions.

Installation

Follow the installation instructions, and view your function's enhanced metrics, traces and logs in Datadog.

Configuration

Follow the configuration instructions to tag your telemetry, capture request/response payloads, filter or scrub sensitive information from logs or traces, and more.

For additional tracing configuration options, check out the official documentation for Datadog trace client.

Besides the environment variables supported by dd-trace-js, the datadog-lambda-js library added following environment variables.

Environment Variables	Description	Default Value
DD_ENCODE_AUTHORIZER_CONTEXT	When set to true for Lambda authorizers, the tracing context will be encoded into the response for propagation. Supported for NodeJS and Python.	true
DD_DECODE_AUTHORIZER_CONTEXT	When set to true for Lambdas that are authorized via Lambda authorizers, it will parse and use the encoded tracing context (if found). Supported for NodeJS and Python.	true
DD_COLD_START_TRACING	Set to false to disable Cold Start Tracing. Used in NodeJS and Python.	true
DD_MIN_COLD_START_DURATION	Sets the minimum duration (in milliseconds) for a module load event to be traced via Cold Start Tracing. Number.	3
DD_COLD_START_TRACE_SKIP_LIB	optionally skip creating Cold Start Spans for a comma-separated list of libraries. Useful to limit depth or skip known libraries.	./opentracing/tracer
DD_CAPTURE_LAMBDA_PAYLOAD	[Captures incoming and outgoing AWS Lambda payloads][1] in the Datadog APM spans for Lambda invocations.	false
DD_CAPTURE_LAMBDA_PAYLOAD_MAX_DEPTH	Determines the level of detail captured from AWS Lambda payloads, which are then assigned as tags for the aws.lambda span. It specifies the nesting depth of the JSON payload structure to process. Once the specified maximum depth is reached, the tag's value is set to the stringified value of any nested elements beyond this level. For example, given the input payload: { "lv1" : { "lv2": { "lv3": "val" } } } If the depth is set to 2, the resulting tag's key is set to function.request.lv1.lv2 and the value is {\"lv3\": \"val\"}. If the depth is set to 0, the resulting tag's key is set to function.request and value is {\"lv1\":{\"lv2\":{\"lv3\": \"val\"}}}	10

Lambda Profiling Beta

Datadog's Continuous Profiler is now available in beta for NodeJS in version 6.87.0 and layer version 87 and above. This optional feature is enabled by setting the DD_PROFILING_ENABLED environment variable to true. During the beta period, profiling is available at no additional cost.

Major Version Notes

5.x.x

The 5.x.x release introduces version 2 of the Datadog tracer, dd-trace-js. This includes a few breaking changes, and the migration guide found here.

The first 5.x.x version was released with Lambda Layer version 69.

6.x.x

The 6.x.x release introduces support for the node 16 runtime and esm modules.

7.x.x

The 7.x.x release drops support for Node12, and upgrades dd-trace-js to version 3.x

9.x.x

The 9.x.x release changed how Lambda's traceID is hashed if the incoming payload contains Step Functions context object. This change only affects those who uses inject Step Functions context object into Lambda payload.

There is a full migration guide available here. Some changes are more likely to impact Serverless users:

• HTTP Operation Name Changed. HTTP requests will no longer appear as a separate client under *-http-client, which polluted the APM service catalog.
• tracer.currentSpan() has been deprecated for a long time, and is now removed.
• tracer.bindEmitter() is similarly removed after being deprecated.
• It is no longer possible to bind promises or event emitters with tracer.scope().bind(...)

Opening Issues

If you encounter a bug with this package, we want to hear about it. Before opening a new issue, search the existing issues to avoid duplicates.

When opening an issue, include the Datadog Lambda Layer version, Node version, and stack trace if available. In addition, include the steps to reproduce when appropriate.

You can also open an issue for a feature request.

Contributing

If you find an issue with this package and have a fix, please feel free to open a pull request following the procedures.

Community

For product feedback and questions, join the #serverless channel in the Datadog community on Slack.

License

Unless explicitly stated otherwise all files in this repository are licensed under the Apache License Version 2.0.

This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2019 Datadog, Inc.

datadog-lambda-js's Issues

Custom traces not forwarded to DataDog when Kinesis subscription used

Background

We created an example repository that uses the Serverless Framework plugin and the Lambda Forwarder to push metrics and DataDog Traces to DataDog. It was based on the standard Serverless Framework TypeScript template sls create --template=aws-nodejs-typescript:

serverless.yaml

service:
  name: datadog-test

custom:
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true
  datadog:
    addLayers: true
    logLevel: "info"
    flushMetricsToLogs: true
    site: datadoghq.eu
    enableXrayTracing: true
    enableDDTracing: false
    enableTags: true

# Add the serverless-webpack plugin
plugins:
  - serverless-webpack
  - serverless-plugin-datadog

provider:
  name: aws
  region: eu-west-1
  runtime: nodejs12.x
  apiGateway:
    minimumCompressionSize: 1024 # Enable gzip compression for responses > 1 KB
  environment:
    AWS_NODEJS_CONNECTION_REUSE_ENABLED: 1

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          method: get
          path: hello

The DataDog trace library was configured to split requests by domain:

tracer.ts

import ddtrace from "dd-trace";
const tracer = ddtrace.init();
tracer.use("http", {
  splitByDomain: true,
});
export default tracer;

The handler makes a couple of HTTP requests, logs a metric and some spans:

handler.ts

import tracer from "./tracer";
import { APIGatewayProxyHandler } from "aws-lambda";
import "source-map-support/register";
import fetch from "node-fetch";
import { datadog, sendDistributionMetric } from "datadog-lambda-js";

export const handler: APIGatewayProxyHandler = async (_event, _context) => {
  const jsonSpan = tracer.startSpan("call_json_placeholder")
  await fetch("https://jsonplaceholder.typicode.com/todos/1");
  jsonSpan.finish()
  try {
    await fetch("https://httpstat.us/500");
  } catch(e) {
    console.log("error calling httpstat.us", e);
  }
  sendDistributionMetric("randomValue", Math.random()); 
  return {
    statusCode: 200,
    body: JSON.stringify({ msg: "ok" }),
  };
};

export const hello = datadog(handler);

We use this Lambda to test that our platform observability is working OK.

It was working as expected, but a compliance requirement then required us to create a subscription on our CloudWatch Log Groups to export them to another service. It's only possible to have a single subscription filter on a CloudWatch Log Group (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html), so the Lambda forwarder stopped executing.

To get around this, we set up a single Kinesis stream following the blog post at https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-kinesis-firehose-destination/?tab=kinesisfirehosedeliverystream

With the Kinesis integration in place, we now have logs coming in to DataDog (and placed in our audit systems), but not DataDog traces.

In practice, this means, that in the above example, the call_json_placeholder span isn't shown at all within DataDog.

We were expecting the Kinesis delivery to be able to handle spans, however, looking at https://github.com/DataDog/datadog-serverless-functions/blob/master/aws/logs_monitoring/lambda_function.py#L91 it looks like the forwarder has some logic built-in to do something special with spans. Perhaps this is missing from the DataDog Kinesis subscription?

What's the correct way to deal with this?

Expected Behavior

Expected the trace details to be shown within DataDog.

Actual Behavior

No trace details shown.

Steps to Reproduce the Problem

1. Create the template as above.
2. Remove the CloudWatch Filter subscription from the Lambda Forwarder and replace with Kinesis stream.
3. Observe that no traces now appear in DataDog.

Specifications

{
  "name": "datadog-test",
  "version": "1.0.0",
  "description": "Serverless webpack example using Typescript",
  "main": "handler.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "dependencies": {
    "@types/node-fetch": "^2.5.7",
    "datadog-lambda-js": "^2.24.0",
    "dd-trace": "^0.22.2",
    "node-fetch": "^2.6.0",
    "source-map-support": "^0.5.10"
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.17",
    "@types/node": "^10.12.18",
    "fork-ts-checker-webpack-plugin": "^3.0.1",
    "serverless": "^1.74.1",
    "serverless-plugin-datadog": "^1.0.1",
    "serverless-webpack": "^5.2.0",
    "ts-loader": "^5.3.3",
    "typescript": "^3.2.4",
    "webpack": "^4.29.0",
    "webpack-node-externals": "^1.7.2"
  },
  "author": "The serverless webpack authors (https://github.com/elastic-coders/serverless-webpack)",
  "license": "MIT"
}

Is it possible to get end to end traces for aws serverless lambda, for all possible triggers otherthan AWS Gateway?

Is it possible to get end to end traces for aws serverless lambda, for all possible triggers otherthan AWS Gateway?

I tested with aws x-ray, it is supporting traces for AWS Gateway and Lambda correlation.

But I was stuck at other possible lambda triggers.

Does datadog agent supports CloudwatchEvents, Log Subscriptions, DynamoDb Streams, S3 file triggers ..etc., triggers to lambda.

I need to derive correlation for all possible lambda triggers.
trigger(API Gateway or Streams or Events..) --------------> Lmabda ---------------> DynamoDB

TypeError - handler is not a function

Expected Behavior

Lambda layer functions correctly

Actual Behavior

As of 5am BST the datadog lambda layer versions 62 and 64 for node 14.x in us-east-1 are intermittently exhibiting a type error on lambda execution. Same layer in eu-west-1 with same code is not exhibiting the issue. The lambda functioned correctly for 18 hours with version 62 of the layer then started getting the type error which persisted for 90 minutes until same code was redeployed with version 64 of the layer. This then worked for 2 hours before giving the same type error and has done so for the last hour.

Steps to Reproduce the Problem

1. Apply the layer on a lambda in us-east-1

Specifications

• Datadog Lambda Layer version: 62 & 64
• Node version: 14.x

Stacktrace

{
    "errorType": "TypeError",
    "errorMessage": "handler is not a function",
    "stack": [
        "TypeError: handler is not a function",
        "    at /opt/nodejs/node_modules/datadog-lambda-js/utils/handler.js:148:25",
        "    at Object.<anonymous> (/opt/nodejs/node_modules/datadog-lambda-js/index.js:168:62)",
        "    at step (/opt/nodejs/node_modules/datadog-lambda-js/index.js:44:23)",
        "    at Object.next (/opt/nodejs/node_modules/datadog-lambda-js/index.js:25:53)",
        "    at /opt/nodejs/node_modules/datadog-lambda-js/index.js:19:71",
        "    at new Promise (<anonymous>)",
        "    at __awaiter (/opt/nodejs/node_modules/datadog-lambda-js/index.js:15:12)",
        "    at /opt/nodejs/node_modules/datadog-lambda-js/index.js:162:108",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:97:56",
        "    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:48:56"
    ]
}

Cloudwatch error after setting up our lambda function with CDK construct

Expected Behavior

We use the datadog-cdk-construct to instrument our lambda functions. Most of our logs look like this:

START RequestId: ba4d56aa-cb23-468a-960b-5df98d09e9df Version: $LATEST
2022-01-05T10:27:51.431Z	ba4d56aa-cb23-468a-960b-5df98d09e9df	INFO	[dd.trace_id=7576945396481994720 dd.span_id=7576945396481994720] DATADOG TRACER CONFIGURATION - {
    "date": "2022-01-05T10:27:51.430Z",
    "os_name": "Linux",
    "os_version": "4.14.252-207.481.amzn2.x86_64",
    "architecture": "x64",
    "version": "1.1.2",
    "lang": "nodejs",
    "lang_version": "14.18.1",
    "enabled": true,
    "service": "...",
    "agent_url": "http://127.0.0.1:8126",
    "debug": false,
    "sample_rate": 1,
    "sampling_rules": [],
    "tags": {
        "_dd.origin": "lambda",
        "service": "..."
    },
    "log_injection_enabled": true,
    "runtime_metrics_enabled": false,
    "integrations_loaded": [
        "dns",
        "net",
        "pg",
        "http",
        "https",
        "http2"
    ]
}
2022-01-05T10:27:52.491Z	ba4d56aa-cb23-468a-960b-5df98d09e9df	INFO	[dd.trace_id=2744288801532801835 dd.span_id=2744288801532801835] Success
END RequestId: ba4d56aa-cb23-468a-960b-5df98d09e9df

And data is pushed correctly to datadog. We would expect no errors showing up.

Actual Behavior

Quite regularly we get errors in the cloudwatch logs of our newly instrumented lambda function that look like this:

LOGS	Name: datadog-agent	State: Subscribed	Types: [platform,function,extension]
2022-01-05T10:26:42.652Z	undefined	ERROR	{
    "status": "error",
    "message": "datadog:api key not configured, see https://dtdg.co/sls-node-metrics"
}
EXTENSION	Name: datadog-agent	State: Ready	Events: [INVOKE,SHUTDOWN]

We do provide API KEY, via apiKeySecretArn (see below). So not quite sure what does this error means.

Steps to Reproduce the Problem

datadog = new Datadog(scope, id, {
      addLayers: true,
      nodeLayerVersion: 66,
      extensionLayerVersion: 15,
      site: "datadoghq.com",
      enableDatadogTracing: true,
      apiKeySecretArn: ...
    });

datadog.addLambdaFunctions([<LAMBDA_FUNCTIONS>]);

All the CDK construct does is to set the DD_API_KEY_SECRET_ARN env variable. It looks like datadog-lambda-js does not behave well when this is set instead of the usual API key.

Specifications

• Datadog Lambda Layer version: 66
• Node version: 14

NB: this bug was already submitted here DataDog/datadog-cdk-constructs#80 and I was advised to move it here.

Custom trace doesn't appear in datadog dashboard

Expected Behavior

Should be able to see custom trace in datadog dashboard.

Actual Behavior

Only x ray tracing spam is appear in datadog even though I can see generated trace logs in cloudwatch.

Steps to Reproduce the Problem

1. Submit custom trace using tracer.trace()

Specifications

• Datadog Lambda Layer version: ^2.24.0
• Node version: v12.16.1

Stacktrace

 "traces": [
      [
          {
              "trace_id": "255c1a7018c7151d",
              "span_id": "13436f6fb3eb0d30",
              "parent_id": "6fa3efa57542847b",
              "name": "http.request",
              "resource": "GET",
              "error": 0,
              "meta": {
                  "_dd.origin": "lambda",
                  "service": "fp-apac-nimbus-menu-service-apollo-dev-updateMenu",
                  "span.kind": "client",
                  "http.method": "GET",
                  "http.url": "https://xxxx/api/v1/catalog/menus",
                  "http.status_code": "200"
              },
              "metrics": {
                  "_sample_rate": 1,
                  "_sampling_priority_v1": 2
              },
              "start": 1605621793470250000,
              "duration": 723025879,
              "service": "fp-apac-nimbus-menu-service-apollo-dev-updateMenu-http-client",
              "type": "http"
          },

Question: Custom logger using Datadog lambda layer

When using the Datadog lambada layer which contains the datadog-lambda-js lib, what is the recommended approach to utilize a custom logger implementation like shown on the README example?

Usage with Lambda@Edge

Expected Behavior

Aggregate logs from multiple regions into DataDog from Lambda@Edge

Actual Behavior

Not sure...not seeing logs. We setup Forwarders in multiple regions but are unsure how / if to use this package in combination with that.

DataDog/datadog-serverless-functions#359

Steps to Reproduce the Problem

For our non Lambda@Edge Lambdas I use the Serverless Plugin Datadog in combination with this datadog-lambda-js library and we see consistent logs / traces in DataDog

When using Lambda@Edge Cloudwatch logs occur in the AWS region where the request is routed to. We have setup forwarders in multiple regions but still are seeing no logs. I tried wrapping my handler in the datadog function and tried variations both with:

process.env.DD_FLUSH_TO_LOG = 'false';

and

process.env.DD_FLUSH_TO_LOG = 'true';
process.env.DD_SITE = 'datadoghq.com';
process.env.DD_API_KEY = dataDogApiKey

Specifications

• Datadog Lambda Layer version: ?
• Node version: 12.x

Possible to not send distribution metrics?

If we're wrapping lambdas that use shared components that are emitting metrics using statsd format of increments/etc, then if we send the same metric name using distribution metrics the metrics conflict in the datadog api.

I recently ran into this issue where we had a lambda temporarily emitting distribution metrics (because we didn't really understand what this is and why its different from standard statsd metrics) and it clobbered our existing metrics graphs.

Is it possible to use datadog-js and use existing statsd formats?

Add support for all custom metrics types

Expected Behavior

I would like to be able to publish all of the custom metric types supported by DataDog:

• COUNT
• RATE
• GAUGE
• HISTOGRAM
• DISTRIBUTION

Right now it's only possible by using outdated and not official DataDog API clients or by using DogStatsD, which obviously cannot be setup in AWS Lambda.

Actual Behavior

I'm not able to select custom metric type using this function:
function sendDistributionMetric(name: string, value: number, ...tags: string[]): void;

Specifications

• Datadog Lambda Layer version: None, NPM package "datadog-lambda-js": "2.17.0"
• Node version: 12.x

Error Finding AWS Credentials

Expected Behavior

Wrap the handler with { datadog } function and see logs when supplying forwarder ARN from

Actual Behavior

Used the example

const messageToObject = (stringMessage) => {
  const { message, status, ...metadata } = JSON.parse(stringMessage); // eslint-disable-line @typescript-eslint/no-unused-vars

  return [metadata, message];
};

const wrapper = (handler) => {
  const logger = pino();

  return datadog(handler, {
    logger: {
      debug: (message) => logger.debug(...messageToObject(message)),
      error: (message) => logger.error(...messageToObject(message)),
    },
  });
};

exports.handler = wrapper(handler);

I'm also using Pino to log messages to STDOUT within the handler, not sure if that's the issue?

Steps to Reproduce the Problem

1. This was working previously and stopped. I'm not sure if something changed in our config but help debugging would be useful.

Specifications

• Datadog Lambda Layer version: ?
• Node version: 12.x

Stacktrace

{"error.msg":"ENOENT: no such file or directory, open '/home/sbx_user1051/.aws/credentials'","error.type":"Error","error.stack":"Error: ENOENT: no such file or directory, open '/home/sbx_user1051/.aws/credentials'\n    at Object.openSync (fs.js:462:3)\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:53\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:56\n    at Scope._activate (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/scope/async_hooks.js:53:14)\n    at Scope.activate (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:12:19)\n    at DatadogTracer.trace (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:35)\n    at Object.openSync (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:23)\n    at Object.readFileSync (fs.js:364:35)\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:53\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:56","_dd.origin":"lambda","service":"discovery-optimizely-fullstack-ab-test-prod-datafile-webhook","component":"fs","span.kind":"internal","file.path":"/var/task/node_modules/aws-xray-sdk-core/lib/middleware/sampling/local_reservoir.js","file.flag":"r"},"metrics":{"_sample_rate":1,"_sampling_priority_v1":1},"start":1601447535843548400,"duration":448730,"service":"discovery-optimizely-fullstack-ab-test-prod-datafile-webhook-fs"},{"trace_id":"2404f81bae32c2eb","span_id":"0d2b6d4b62711afa","parent_id":"2c970dd2db1d990a","name":"fs.operation","resource":"openSync","error":0,"meta":{"error.msg":"ENOENT: no such file or directory, open '/home/sbx_user1051/.aws/config'","error.type":"Error","error.stack":"Error: ENOENT: no such file or directory, open '/home/sbx_user1051/.aws/config'\n    at Object.openSync (fs.js:462:3)\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:53\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:56\n    at Scope._activate (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/scope/async_hooks.js:53:14)\n    at Scope.activate (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:12:19)\n    at DatadogTracer.trace (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:35)\n    at Object.openSync (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:23)\n    at Object.readFileSync (fs.js:364:35)\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:53\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:56","_dd.origin":"lambda","service":"discovery-optimizely-fullstack-ab-test-prod-datafile-webhook","component":"fs","span.kind":"internal","file.path":"/var/task/node_modules/aws-xray-sdk-core/lib/middleware/sampling/local_reservoir.js","file.flag":"r"},"metrics":{"_sample_rate":1,"_sampling_priority_v1":1},"start":1601447535849416700,"duration":176270,"service":"discovery-optimizely-fullstack-ab-test-prod-datafile-webhook-fs"},{"trace_id":"2404f81bae32c2eb","span_id":"2c970dd2db1d990a","parent_id":"5e5ecd342765cae7","name":"fs.operation","resource":"readFileSync","error":0,"meta":{"error.msg":"ENOENT: no such file or directory, open '/home/sbx_user1051/.aws/config'","error.type":"Error","error.stack":"Error: ENOENT: no such file or directory, open '/home/sbx_user1051/.aws/config'\n    at Object.openSync (fs.js:462:3)\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:53\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:56\n    at Scope._activate (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/scope/async_hooks.js:53:14)\n    at Scope.activate (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:12:19)\n    at DatadogTracer.trace (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:35)\n    at Object.openSync (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:23)\n    at Object.readFileSync (fs.js:364:35)\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:93:53\n    at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:45:56","_dd.origin":"lambda","service":"discovery-optimizely-fullstack-ab-test-prod-datafile-webhook","component":"fs","span.kind":"internal","file.path":"/var/task/node_modules/aws-xray-sdk-core/lib/middleware/sampling/local_reservoir.js","file.flag":"r"}

Support for NodeJS lambda profiling

Expected Behavior

See profiling for NodeJS in lambda

Actual Behavior

Profiling tab remains empty

Steps to Reproduce the Problem

1. Setup APM for serverless NodeJS.
2. Enable DD_PROFILING_ENABLED

Specifications

• Datadog Lambda Layer version: 66
• Node version: 14.x

DATADOG TRACER CONFIGURATION - 
{
    "date": "2021-12-01T12:08:03.119Z",
    "os_name": "Linux",
    "os_version": "4.14.246-198.474.amzn2.x86_64",
    "architecture": "x64",
    "version": "1.5.1",
    "lang": "nodejs",
    "lang_version": "14.18.1",
    "enabled": true,
    "service": "NodeAxiosDataDogExample",
    "agent_url": "http://127.0.0.1:8126",
    "debug": false,
    "sample_rate": 1,
    "sampling_rules": [],
    "tags": {
        "_dd.origin": "lambda",
        "service": "NodeAxiosDataDogExample",
        "runtime-id": "5c661f49-9c7d-435d-a3ea-ecd1954bf3ce"
    },
    "log_injection_enabled": false,
    "runtime_metrics_enabled": false,
    "profiling_enabled": true,
    "integrations_loaded": [
        "http",
        "https"
    ]
}

DD_SERVICE not showing in APM service list

Expected Behavior

Expected behaviour is for "servicename" to appear in our service list in APM.

Actual Behavior

"servicename" appears as a resource of the aws.lambda service.

Documentation is contradictory and not clear (https://docs.datadoghq.com/tracing/serverless_functions/?tab=nodejs#organizing-your-serverless-infrastructure-with-tags):

By default, each Lambda function is treated as its own service. Add your own tag to override this.

Note: The default behavior for new Datadog customers is for all Lambda functions to be grouped under the aws.lambda service, and represented as a single node on the Service map. Tag your functions by service to override this.

Steps to Reproduce the Problem

Our lambda function is instrumented with the datadog-lambda-js library, configured with DD_SERVICE="servicename" and shipped using aws-dd-forwarder-3.14.0.zip. The forwarder is deployed manually and configured with the DD_FETCH_LAMBDA_TAGS=true.

A variant that I tried was to use DD_TAGS=service:servicename.

"servicename" is using the library handler instead of the Lambda layer.

Specifications

• Datadog Lambda Layer version: "datadog-lambda-js": "^3.29.0"
• Node version: Node.js 12.x

Issue with DD_KMS_API_KEY

Expected Behavior

I would expect metrics to be recorded on Datadog when using the DD_KMS_API_KEY variable.

Actual Behavior

No metrics are being recorded.

Some times I get this log output (but not all of the times):

START RequestId: 8d48efe8-8a70-45fe-87fd-92ed37598d1e Version: $LATEST
{"message":"[dd.trace_id=6021779273530932769 dd.span_id=5330958935018271181] {\"innerError\":{\"message\":null,\"code\":\"InvalidCiphertextException\",\"time\":\"2020-10-20T12:38:23.593Z\",\"requestId\":\"8ed89fe0-f9b7-47d4-a4aa-b6ba896e3299\",\"statusCode\":400,\"retryable\":false,\"retryDelay\":13.137550140158893},\"status\":\"error\",\"message\":\"datadog:couldn't decrypt kms api key\"}","level":"error","severity":"ERROR","timestamp":"2020-10-20T12:38:23.613Z"}
END RequestId: 8d48efe8-8a70-45fe-87fd-92ed37598d1e
REPORT RequestId: 8d48efe8-8a70-45fe-87fd-92ed37598d1e	Duration: 1148.02 ms	Billed Duration: 1200 ms	Memory Size: 256 MB	Max Memory Used: 198 MB	Init Duration: 1833.00 ms	

Steps to Reproduce the Problem

1. Encrypt an API Key from Datadog using the helpers in the AWS Lambda Console.
2. Add the appropriate policy to Lambdas' role as instructed by AWS Lambda Console.
3. Try to run the Lambda function

Specifications

• Datadog Lambda Layer version: latest
• Node version: 12

Feature Request: Support a less invasive way of instrumenting lambdas

Expected Behavior

I can set AWS_LAMBDA_EXEC_WRAPPER pointing to DD Lambda layer without touching anything else in the Lambda configuration in order to get tracing.

Here's an example of a similar tracing solution (OpenTelemetry): https://github.com/open-telemetry/opentelemetry-lambda/blob/main/nodejs/README.md

Actual Behavior

The current behavior is a terrible DX choice IMO:

• I have to change my main lambda handler to /opt/nodejs/node_modules/datadog-lambda-js/handler.handler
• Then I can configure DD_LAMBDA_HANDLER with my actual handler function

Unable to install package due to bad version on hot-shots dependency

The issue is an invalid version in the packages.json file:

hot-shots@true7.7.1

should be

[email protected]

Expected Behavior

npm install should install the package

Actual Behavior

npm install fails to install the package with this error:

Steps to Reproduce the Problem

1. run npm install

Specifications

• Datadog Lambda Layer version: N/A
• Node version: 12.19.0

Stacktrace

npm ERR! code ETARGET
npm ERR! notarget No matching version found for [email protected].
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of 'datadog-lambda-js'
npm ERR! notarget

Need to close socket

datadog-lambda-js/src/trace/context.ts

Line 134 in b968303

const client = createSocket("udp4");

Socket is opened but never closed. This causes our application host to run out of file descriptors. We need to restart the host to recover lost resources.

Datadog lambda layer version:
"datadog-lambda-js": "^3.27.0"

Node version:
12.x (whatever AWS lambda uses)

Error logs:

{
    "error": {
        "message": "getaddrinfo EMFILE dynamodb.us-east-1.amazonaws.com",
        "errno": "EMFILE",
        "code": "NetworkingError",
        "syscall": "getaddrinfo",
        "hostname": "dynamodb.us-east-1.amazonaws.com",
        "region": "us-east-1",
        "retryable": true,
        "time": "2020-10-15T09:34:05.022Z"
    },
    "request_id": "be84c269-b2eb-4fbc-82bf-202c9e12ec62",
    "stack_trace": [
        {
            "function_name": "processTicksAndRejections",
            "file": {
                "name": "internal/process/task_queues.js",
                "location": {
                    "line": 97,
                    "column": 5
                }
            }
        }
    ],
    "dd": {
        "trace_id": "2308384582283572634",
        "span_id": "6702646418954244696",
        "service": "xxx",
        "env": "xxx"
    }
}

Bug: sendDistributionMetricWithDate custom date doesn't have any effect in DataDog

Expected Behavior

When I send data point using sendDistributionMetricWithDate function, I want to see in DataDog data point with matching timestamp.

Actual Behavior

When I send data point using sendDistributionMetricWithDate function, DataDog ignores date provided and displays data point with the ingestion time.

Steps to Reproduce the Problem

1. Call sendDistributionMetricWithDate with Date from the past, for example 2 hours behind.
2. Wait for data point to appear in DataDog.

Specifications

• Datadog Lambda Layer version: "datadog-lambda-js": "2.20.0"
• Node version: 12.x

More details

I observed the same behaviour for setting 30 minutes in the past and 10 minutes into the future. All of the data points were displayed in DataDog with ingestion date instead of specified date.

Logs and screenshots

Log from Lambda in debug mode:

2020-04-02T12:44:57.304Z	375befed-1693-4bd6-9fd4-45139104c9b9	DEBUG	[dd.trace_id=5038852051485810583 dd.span_id=1784722096679938107] 
{
    "status": "debug",
    "message": "datadog:sending payload with body {\"series\":[{\"metric\":\"mux.test.viewer_experience_score\",\"points\":[[1585821600545,[0.0]]],\"tags\":[\"dd_lambda_layer:datadog-nodev12.16.1\"],\"type\":\"distribution\"}]}"
}

Timestamp is in UTC:

> new Date(1585821600545)
2020-04-02T10:00:00.545Z

Parsed JSON:

{
  "series": [
    {
      "metric": "mux.test.viewer_experience_score",
      "points": [
        [
          1585821600545,
          [
            0.0
          ]
        ]
      ],
      "tags": [
        "dd_lambda_layer:datadog-nodev12.16.1"
      ],
      "type": "distribution"
    }
  ]
}

I was looking as far as 15 months into the past but couldn't find any data points.

I believe something is wrong with the API, since the request according to the logs looks completely fine.

Allow set custom timestamp for metric datapoint

Expected Behavior

When I'm sending datapoint with sendDistributionMetric function, I would like to specify my custom timestamp.

Actual Behavior

Right now it automatically uses the actual time for timestamp generation.

Specifications

• Datadog Lambda Layer version: none, datadog-lambda-layer-js:2.17.0
• Node version: 12.x

sendDistributionMetric() syntax error using Lambda Layer

Expected Behavior

Using the example code from the Node section of this page. Removing the call to sendDistributionMetric() results in no syntax error.

Actual Behavior

Syntax error is returned.

Steps to Reproduce the Problem

1. Use the installing and using the DataDog layer example to create a new lambda with the arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Node10-x:3 layer
2. Set the DD_API_KEY env var
3. Set the DD_FORWARD_LOG to false.

Specifications

• Datadog Lambda Layer version: 3
• Node version: 10.x

Stacktrace

{
  "errorType": "Runtime.UserCodeSyntaxError",
  "errorMessage": "SyntaxError: Unexpected token {",
  "trace": [
    "Runtime.UserCodeSyntaxError: SyntaxError: Unexpected token {",
    "    at _loadUserApp (/var/runtime/UserFunction.js:98:13)",
    "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
    "    at Object.<anonymous> (/var/runtime/index.js:36:30)",
    "    at Module._compile (internal/modules/cjs/loader.js:776:30)",
    "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)",
    "    at Module.load (internal/modules/cjs/loader.js:653:32)",
    "    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)",
    "    at Function.Module._load (internal/modules/cjs/loader.js:585:3)",
    "    at Function.Module.runMain (internal/modules/cjs/loader.js:829:12)",
    "    at startup (internal/bootstrap/node.js:283:19)"
  ]
}

Webpack not able to bundle datadog-lambda-js

Expected Behavior

datadog-lambda-js handler doesn't throw any errors

Actual Behavior

webpack is not able to package datadog-lambda-js handler due to dependencies on lambda runtime.

Steps to Reproduce the Problem

1. Follow the steps mentioned here. https://docs.datadoghq.com/serverless/installation/nodejs/?tab=custom#using-the-package
2. Run serverless package command.
3. Webpack fails with an error.

Specifications

• Datadog Lambda Layer version: N.A
• Node version: 14.x

Stacktrace

ERROR in ./node_modules/datadog-lambda-js/dist/handler.js 8:11-52
Module not found: Error: Can't resolve '/var/runtime/UserFunction' in '/Users/mohitm/dev/git/serverless-request-management/node_modules/datadog-lambda-js/dist'
resolve '/var/runtime/UserFunction' in '/Users/mohitm/dev/git/serverless-request-management/node_modules/datadog-lambda-js/dist'
using description file: /Users/mohitm/dev/git/serverless-request-management/node_modules/datadog-lambda-js/package.json (relative path: ./dist)
  root path /Users/mohitm/dev/git/serverless-request-management
    using description file: /Users/mohitm/dev/git/serverless-request-management/package.json (relative path: ./var/runtime/UserFunction)
      no extension
        /Users/mohitm/dev/git/serverless-request-management/var/runtime/UserFunction doesn't exist
      .js
        /Users/mohitm/dev/git/serverless-request-management/var/runtime/UserFunction.js doesn't exist
      .json
        /Users/mohitm/dev/git/serverless-request-management/var/runtime/UserFunction.json doesn't exist
      .wasm
        /Users/mohitm/dev/git/serverless-request-management/var/runtime/UserFunction.wasm doesn't exist
      as directory
        /Users/mohitm/dev/git/serverless-request-management/var/runtime/UserFunction doesn't exist
  No description file found in /var/runtime or above
  no extension
    /var/runtime/UserFunction doesn't exist
  .js
    /var/runtime/UserFunction.js doesn't exist
  .json
    /var/runtime/UserFunction.json doesn't exist
  .wasm
    /var/runtime/UserFunction.wasm doesn't exist
  as directory
    /var/runtime/UserFunction doesn't exist

webpack compiled with 1 error and 2 warnings in 366 ms

Webpack config

const slsw = require('serverless-webpack');
const CopyPlugin = require('copy-webpack-plugin');
const nodeExternals = require('webpack-node-externals');

module.exports = {
  externals: ['dd-trace', 'datadog-lambda-js', nodeExternals()],
  externalsPresets: {
    node: true
  },
  optimization: {
    minimize: false
  },
  entry: slsw.lib.entries,
  target: 'node',
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  devtool: 'inline-cheap-module-source-map'
};

Webpack serverless config

webpack:
    webpackConfig: 'webpack.config.js' # Name of webpack configuration file
    packager: 'npm' # Packager that will be used to package your external modules
    excludeFiles: tests/ # Provide a glob for files to ignore
    concurrency: 6
    serializedCompile: true
    includeModules:
      forceExclude:
        - dd-trace
        - datadog-lambda-js

function definition in serverless.yml

functions:
  auth:
    handler: node_modules/datadog-lambda-js/dist/handler.handler
    environment:
      DD_LAMBDA_HANDLER: functions/auth.handler
    timeout: 30

No matching version found for [email protected]

Expected Behavior

npm install datadog-lambda-js should work

Actual Behavior

npm install datadog-lambda-js                                                                                                                     
npm ERR! code ETARGET
npm ERR! notarget No matching version found for [email protected].
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget 
npm ERR! notarget It was specified as a dependency of 'datadog-lambda-js'
npm ERR! notarget 

I think there is invalid depdency on serialize-error Ref

    "serialize-error": "true7.0.1",

maybe this instead ?

    "serialize-error": "^7.0.1",

Steps to Reproduce the Problem

1. npm install datadog-lambda-js

Specifications

• Datadog Lambda Layer version: latest
• Node version: v12.16.1

API Gateway v1 protocol causes 500 response from API gateway (possible regression in 3.46.0)

Expected Behaviour

Given an API Gateway integration with the v1 protocol in use, a simple Lambda function should return 200 OK on a valid route.

Actual Behaviour

API gateway returned 500. With no change to any other configuration, changing the API Gateway to use protocol version 2 fixed the issue.

Using the function handler directly (without Datadog), the same function code responds with 200 OK for both protocol versions.

Steps to Reproduce the Problem

1. Create a Lambda with the following handler code:

   module.exports.handler = async event => {
     return {
         "isBase64Encoded": false,
         "statusCode": 200,
         "headers": { "Content-Type": "application/json" },
         "multiValueHeaders": { },
         "body": "{}"
     }
   }

2. Configure the Lambda with the v3.46.0 layer, and the appropriate handler and environment config.
3. Connect the created Lambda to an HTTP API Gateway with the integration configured to use protocol version 1.

Specifications

• Datadog Lambda Layer version: 3.46.0
• Node version: 12

Workaround

Downgrade the layer to the previous release (v3.44.0)

Custom Metrics

Hi,

I'd like to use a scheduled lambda to create some custom metrics. As I understand it, I can use this library to give me metrics about that lambda's execution, but I cannot use it to create custom metrics such as a gauge. Could you expose the current metric listener so that we can submit custom metrics?

Thanks

Unable to control service name of wrapped handlers

Expected Behavior

I'd like to be able to control the name of the service that appears in my facet list for the function that is traced. I've been unable to find a way to control this and its always named the function. My service is made up of many functions so this isn't very meaningful and adds junk to my facet list.

Thanks!

Additional metric types

Is there any technical reason why is only the distribution metric implemented and not other like COUNT or HISTOGRAM? Using DISTRIBUTION is quite wasteful (i.e. it uses 5x more custom metrics) if I only need to count things.

Cannot find redirected handler

Expected Behavior

Adding the layer to my lambda, setting up the required environment variables, and changing my lambda handler to /opt/nodejs/node_modules/datadog-lambda-js/handler.handler will handle tracing automatically for me.

Actual Behavior

In the console I have this message

Lambda can't find the file /opt/nodejs/node_modules/datadog-lambda-js/handler.js. Make sure that your handler upholds the format: file-name.method.

When I send a test event, some expected things happen:

• trace appears in APM
• dd.trace_id & dd.span_id appear in the logs and I can filter by them

However, the maximum memory limit is reached 100% of the time (128MB in this case).

Steps to Reproduce the Problem

1. Create a lambda with the DD layer
2. Send a test event

This is the relevant portion of the terraform defining this function:

resource "aws_lambda_function" "lambda-test" {
  function_name = "my-lambda-test"
  role          = var.iam_role.arn

  handler = "/opt/nodejs/node_modules/datadog-lambda-js/handler.handler"
  runtime = "nodejs12.x"
  timeout     = 60
  memory_size = 128

  environment {
    variables = merge(
      {
        DD_TRACE_ENABLED    = true,
        DD_API_KEY          = var.datadog_api_key
        DD_LAMBDA_HANDLER   = "src/index.handler",
        DD_LOG_LEVEL        = "debug",
        DD_LOGS_INJECTION   = true
        DD_FLUSH_TO_LOG     = true
        DD_ENHANCED_METRICS = false
      },
      var.environment_vars
    )
  }

  depends_on = [
    aws_cloudwatch_log_group.lambda_logs
  ]

  layers = ["arn:aws:lambda:eu-west-1:464622532012:layer:Datadog-Node12-x:24"]
}

Specifications

• Datadog Lambda Layer version: arn:aws:lambda:eu-west-1:464622532012:layer:Datadog-Node12-x:24
• Node version: Node.js 12.x

Stacktrace

This is the log output from sending a test event.

er.js:43:35)\n    
at Object.openSync (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:91:23)\n    
at Object.readFileSync (fs.js:360:35)\n    
at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:91:53\n    
at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:43:56",
"service":"my-test-lambda",
"component":"fs",
"span.kind":"internal",
"file.path":"/home/sbx_user1051/.aws/config",
"file.flag":"r"
},
"metrics":{"_sample_rate":1,"_sampling_priority_v1":1},
"start":1592414079831124000,
"duration":180420,
"service":"my-test-lambda-fs"
},{
"trace_id":"312ccbad2fe3fb80",
"span_id":"55dfc3d99a865658",
"parent_id":"0f603fb5c2d0e7ca",
"name":"fs.operation",
"resource":"readFileSync",
"error":0,
"meta":{
  "error.msg":"ENOENT: no such file or directory, open '/home/sbx_user1051/.aws/config'",
  "error.type":"Error",
  "error.stack":"Error: ENOENT: no such file or directory, open '/home/sbx_user1051/.aws/config'\n    
  	at Object.openSync (fs.js:458:3)\n    
  	at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:91:53\n    
  	at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:43:56\n    
  	at Scope._activate (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/scope/async_hooks.js:51:14)\n    
  	at Scope.activate (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/scope/base.js:12:19)\n    
  	at DatadogTracer.trace (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:43:35)\n    
  	at Object.openSync (/opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:91:23)\n    
  	at Object.readFileSync (fs.js:360:35)\n    
  	at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:91:53\n    
  	at /opt/nodejs/node_modules/dd-trace/packages/dd-trace/src/tracer.js:43:56",
  "service":"my-test-lambda",
  "component":"fs",
  "span.kind":"internal",
  "file.path":"/home/sbx_user1051/.aws/config",
  "file.flag":"r"
},
"metrics":{"_sample_rate":1,"_sampling_priority_v1":1},
  "start":1592414079831086300,
  "duration":230469,
  "service":"my-test-lambda-fs"
},{
"trace_id":"312ccbad2fe3fb80",
"span_id":"34b6e0c824c62f0c",
"parent_id":"0f603fb5c2d0e7ca",
"name":"http.request",
"resource":"POST",
"error":0,
"meta":{
  "service":"my-test-lambda",
  "span.kind":"client","http.method":"POST",
  "http.url":"http://sqs.eu-west-1.amazonaws.com:443/",
  "http.status_code":"200"
},
"metrics":{"_sample_rate":1,"_sampling_priority_v1":1},
"start":1592414079831647700,
"duration":102364502,
"service":"my-test-lambda-http-client",
"type":"http"
},{
"trace_id":"312ccbad2fe3fb80",
"span_id":"0f603fb5c2d0e7ca",
"parent_id":"312ccbad2fe3fb80",
"name":"aws.request",
"resource":"sendMessageBatch https://sqs.eu-west-1.amazonaws.com/123456789012/development_my-test-lambda-data",
"error":0,
"meta":{
  "service":"my-test-lambda",
  "span.kind":"client",
  "aws.operation":"sendMessageBatch",
  "aws.region":"eu-west-1",
  "aws.service":"SQS",
  "component":"aws-sdk",
  "aws.response.request_id":"3079362f-3511-58ec-9f80-614b64aee5c3",
  "aws.sqs.queue_name":"https://sqs.eu-west-1.amazonaws.com/123456789012/development_my-test-lambda-data"
},
"metrics":{"_sample_rate":1,"_sampling_priority_v1":1},
"start":1592414079830559000,
"duration":104427246,
"service":"my-test-lambda-aws-sqs"
},{
"trace_id":"312ccbad2fe3fb80",
"span_id":"312ccbad2fe3fb80",
"parent_id":"0000000000000000",
"name":"aws.lambda",
"resource":"src/index.handler",
"error":0,
"meta":{
  "service":"my-test-lambda",
  "cold_start":"false",
  "function_arn":"arn:aws:lambda:eu-west-1:123456789012:function:my-test-lambda",
  "request_id":"db45a3f2-fd7b-40ea-809e-ac1981b4a54c",
  "resource_names":"my-test-lambda",
  "language":"javascript"
},
"metrics":{"_sample_rate":1,"_dd.agent_psr":1,"_sampling_priority_v1":1},
"start":1592414079790683100,
"duration":144564209,
"service":"my-test-lambda",
"type":"serverless"
}]]}
END RequestId: db45a3f2-fd7b-40ea-809e-ac1981b4a54c
REPORT RequestId: db45a3f2-fd7b-40ea-809e-ac1981b4a54c	Duration: 180.87 ms	Billed Duration: 200 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	

Webpack bundling is not able to resolve dependencies statically

Expected Behavior

No warnings in webpack bundling process.

Actual Behavior

Warnings in bundle output:

WARNING in ./node_modules/datadog-lambda-js/dist/trace/tracer-wrapper.js 33:23-30
Critical dependency: require function is used in a way in which dependencies cannot be statically extracted
 @ ./node_modules/datadog-lambda-js/dist/trace/listener.js
 @ ./node_modules/datadog-lambda-js/dist/trace/index.js
 @ ./node_modules/datadog-lambda-js/dist/index.js

WARNING in ./node_modules/datadog-lambda-js/dist/trace/tracer-wrapper.js 34:26-39
Critical dependency: the request of a dependency is an expression
 @ ./node_modules/datadog-lambda-js/dist/trace/listener.js
 @ ./node_modules/datadog-lambda-js/dist/trace/index.js
 @ ./node_modules/datadog-lambda-js/dist/index.js

Steps to Reproduce the Problem

Have library imported in code:

import { datadog, sendDistributionMetric } from 'datadog-lambda-js'

And then trying to make webpack bundle

So far not observed influence on running application, but warnings are printed each time.

This happens because of dynamic module names could be used in require so webpack cant resolve them when building bundle form all sources linked together.

null `headers` from API Gateway causes handler to return null

Expected Behavior

When a Lambda is invoked with an API Gateway REST proxy input payload with null headers it should return the Lambda's response. null headers is the basic behaviour when using the API Gateway test client where no headers are specified. Without the DD wrapper the response is passed through as expected.

{
    "event": {
        ...
        "headers": null,
        ...
    }
}

Actual Behavior

https://github.com/DataDog/datadog-lambda-js/blob/main/src/trace/context.ts#L260 is not guarded properly (see the caller) because typeof null === 'object' in JS which causes an exception during

datadog-lambda-js/src/index.ts

Lines 116 to 124 in 94c9361

	try {
	await traceListener.onStartInvocation(event, context);
	await metricsListener.onStartInvocation(event);
	if (finalConfig.enhancedMetrics) {
	incrementInvocationsMetric(metricsListener, context);
	}
	} catch (err) {
	logDebug(`Failed to start listeners with error ${err}`);
	}

For a reason I can't immediately identify this scenario leads to null returning from the Lambda (which the runtime will return if the handler returned undefined which I suspect is happening).

Steps to Reproduce the Problem

1. Lambda wrapped in DD (can statically return 200 or whatever)
2. Invoke with API Gateway payload including "headers": null
3. Observe null response
4. Change to "headers": {}
5. Observe proper response

Specifications

• Datadog Lambda Layer version: 50
• Node version: 12.x

DataDog Layer doesn't support arm architecture

Expected Behavior

The Lambda to work with datadog and ARM architecture.

Actual Behavior

Lambda deployed using ARM architecture doesn't work with datadog layer.

Steps to Reproduce the Problem

1. Deploy lambda using ARM architecture.

Specifications

• Datadog Lambda Layer version: 65
• Node version: 14.x

Stacktrace

RequestId: $$$$ Error: fork/exec /opt/extensions/datadog-agent: exec format error
Extension.LaunchError

Disable logging to DataDog for local invocation

Expected Behavior

I'd like to be able to locally invoke my function using the Serverless framework and just see DEBUG logs without sending to data dog. Maybe this is possible with existing env variables for configuration but I didn't see these options.

I'm using this library in combination with serverless-plugin-datadog to supply the Forwarder ARN.

Actual Behavior

serverless invoke local -f <my_function> -p ./event.json

See error below

Steps to Reproduce the Problem

1. run a local invocation

Specifications

• Datadog Lambda Layer version: Not sure..where do I find this?
• Node version: 12

Stacktrace

{"status":"error","message":"datadog:api key not configured, see https://dtdg.co/sls-node-metrics"}
{"level":50,"time":1597887273985,"pid":17773,"hostname":"David-Fox-Powell-Pro","msg":"datadog:couldn't read xray trace header from env"}
{"level":50,"time":1597887273996,"pid":17773,"hostname":"David-Fox-Powell-Pro","innerError":{"name":"TypeError","message":"Cannot read property 'split' of undefined","stack":"TypeError: Cannot read property 'split' of undefined\n    at parseLambdaARN (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/utils/arn.ts:22:24)\n    at Object.parseTagsFromARN (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/utils/arn.ts:54:35)\n    at getEnhancedMetricTags (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/metrics/enhanced-metrics.ts:49:15)\n    at incrementEnhancedMetric (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/metrics/enhanced-metrics.ts:74:5)\n    at Object.incrementInvocationsMetric (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/metrics/enhanced-metrics.ts:79:3)\n    at wrappedFunc.<computed> (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/index.ts:109:9)\n    at Object.<anonymous> (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/utils/handler.ts:21:13)\n    at step (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/dist/utils/handler.js:33:23)\n    at Object.next (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/dist/utils/handler.js:14:53)\n    at /Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/dist/utils/handler.js:8:71\n    at new Promise (<anonymous>)\n    at __awaiter (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/dist/utils/handler.js:4:12)\n    at /Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/utils/handler.ts:19:48\n    at exports.handler (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/functions/edge/entry.js:19:26)\n    at /Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:841:30\n    at Promise._execute (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/debuggability.js:384:9)\n    at Promise._resolveFromExecutor (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:518:18)\n    at new Promise (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:103:10)\n    at AwsInvokeLocal.invokeLocalNodeJs (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:794:12)\n    at AwsInvokeLocal.invokeLocal (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:202:19)\n    at AwsInvokeLocal.tryCatcher (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromiseCtx (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:641:10)\n    at _drainQueueStep (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/async.js:97:12)\n    at _drainQueue (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues [as _onImmediate] (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (internal/timers.js:456:21)\n    at process.topLevelDomainCallback (domain.js:137:15)"},"msg":"datadog:Pre-lambda hook threw error"}
{"level":50,"time":1597887275874,"pid":17773,"hostname":"David-Fox-Powell-Pro","innerError":{},"msg":"datadog:failed to flush metrics"

Support DD_API_KEY_SECRET_ARN

Expected Behavior

I would expect there to be feature parity between the python & nodejs lambda layers.

https://pypi.org/project/datadog-lambda/

if I provide a SecretsManager ARN in a variable called DD_API_KEY_SECRET_ARN, the lambda layer should be able to use the credentials found within this resource.

Actual Behavior

not implemented

Timeout when writing function that never returns

Expected Behavior

When writing a handler that never returns a value, using either the async await method or the callback approach, the wrapper will block the function from completing. This doesn't happen when the wrapper isn't used.

Actual Behavior

Function times out

exports.handler = datadog(function (event, context) { });
// Timeout occurs

As a work around, prepending async to the function fixes this

exports.handler = datadog(async function (event, context) { });
// No timeout

Specifications

• Datadog Lambda Layer version: 5 +
• Node version: 10

Support Nodejs12.X

AWS supports nodejs 12.13.0
https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

I'd like there to be a datadog lambda layer that is built for this runtime.

A BadRequest response from the API logs `authorization failed...`

Expected Behavior

On a BadParam Error from the API, don't log that there is an authorization failure.

Actual Behavior

On a BadParam Error from the API we log the following line https://github.com/DataDog/datadog-lambda-js/blob/main/src/metrics/api.ts#L28 - as this check puts the response in the same bucket https://github.com/DataDog/datadog-lambda-js/blob/main/src/utils/request.ts#L56.

So the request is rejected with a BadAuth

Steps to Reproduce the Problem

0. Turn on debug mode
1. Send a malformed metric
2. Check the logs

Specifications

• Datadog Lambda Layer version: 3.49.1
• Node version: 14.x

Yarn Berry + Webpack = datadog-lambda-js cannot require `aws-sdk`

Expected Behavior

datadog-lambda-js should be able to require() aws-sdk in the KMS Service class.

Actual Behavior

Currently see this error on function startup:

datadog:optional dependency aws-sdk not installed. KMS key decryption will not work

Steps to Reproduce the Problem

It's difficult to reproduce this particular error, but hopefully I can explain it best I can in the comments.

Yarn berry provides us with a mapfile, that we must import into our code, to allow the require() function to resolve external dependencies (like lodash, etc.). We import it like require('/path/to/mapfile.js').setup() at the start of the bundled script, which allows the require module resolution to work as usual.

We also bundle our external dependencies as a lambda layer to avoid bloating the size of our bundle. We do not provide the aws-sdk package in our lambda layer as AWS already provides it /var/runtime/node_modules/aws-sdk.

With these things taken into account, we've started noticing that Datadog's Lambda layer complains about not being able to import the aws-sdk, even though I can see that a require for the SDK later on in our code works just fine.

Specifications

• Datadog Lambda Layer version: Datadog-Node14-x, version 52
• Node version: 14

Datadog adds 250ms to lambda duration

Expected Behavior

I'd expect something in the xx ms range.

Actual Behavior

Sending a simple metric adds ~250ms on average to execution.

Steps to Reproduce the Problem

As simple as this

export const hello = datadog(async (event, context) => {
    sendDistributionMetric("custom_events.hello", 1);
    return {
      statusCode: 200,
      headers: {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Credentials': true,
      },
      body: JSON.stringify({
        message: 'Go Serverless Webpack (Typescript) v1.0! Your function executed successfully!',
        input: event,
      }, null, 2),
    };
})

with the following serverless definition

hello:
    handler: handler.hello
    layers:
      - arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Node10-x:17
    events:
      - http:
          method: get
          path: hello
          cors: true

Specifications

• Datadog Lambda Layer version: latest
• Node version: 10x

Automatic trace id log injection doesn't work with pino

Expected Behavior

The documentation states that the trace id should get automatically injected to the logs for correlation if one is using a supported log library. Pino is listed as a supported library.

Log lines that show up in Datadog should have the trace tab enabled and showing the trace for that request.

Actual Behavior

The START, REPORT and END log-lines that are automatically generated by AWS Lambda have the trace tab enabled and the trace showing, but any manually sent log lines from inside the function don't have any trace id associated with them.

Steps to Reproduce the Problem

We use TypeScript and our logger module looks like this:

import * as pino from 'pino'
import { getConfig } from '../config'

const config = getConfig()

export const logger = pino({
  name: 'service-name',
  level: config.LOG_LEVEL,
  formatters: {
    level(level) {
      return { level }
    }
  }
})

It gets used like this:

import { logger } from '../utils/logger'
...
} catch (error) {
      logger.error(error, `Product error`)
      return null
    }

Specifications

• Datadog Lambda Layer version: 49
• Node version: nodejs12.x
• Lambdas written in TypeScript
• Traces are being sent from the lambda and we can see those in the web console, but they are not linked to any log-lines
• console.log gets the trace id

test - please delete

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

• Datadog Lambda Layer version:
• Node version:

Stacktrace

Paste here

How to debug "datadog:handler not initialized"

Expected Behavior

Not getting this error

Actual Behavior

Getting datadog:handler not initialized logged in my lambdas. Whats weird is that I am getting metrics though. I don't care about traces or anything else, I just wanted to send custom metrics to datadog from the lambda.

Also the lambda got significantly slower pulling this layer in and emitting distribution metrics. Is that because its timing out trying to connect to something? I would have expected that the extension running as a process sidecar should make it so that the lambda can execute as fast as normal since its emitting async to the sidecar?

Look at the resulting graphs when I deployed the distrubtion metrics to my lambda

Steps to Reproduce the Problem

1. I have the datadog extension layer set up for my lambda and my lambdas wrapped with datadog(.. my handler...). I am not using the setup of

Set your function’s handler to /opt/nodejs/node_modules/datadog-lambda-js/handler.handler if using the layer, or node_modules/datadog-lambda-js/dist/handler.handler if using the package.
Set the environment variable DD_LAMBDA_HANDLER to your original handler, for example, myfunc.handler.

Because I am webpacking the lambda into a single file, and couldn't figure out how to get the handler to be executed properly since webpack flattens it all. Not sure if that matters or not

Specifications

• Datadog Lambda Layer version: 9
• Node version: 14

Please fix DeprecationWarning caused by Buffer initialization

https://github.com/DataDog/datadog-lambda-js/blob/main/src/trace/context.ts#L165

This line will generate the DeprecationWarning: "Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead."

This warning will be printed as error by Node.js 14 Runtime, so it'll generate a lot of error log events in Datadog. Hope you can fix this issue in the future version. Thanks a lot.

Cannot use datadog-lambda-js with Webpack

Expected Behavior

Bundling datadog-lambda-js works.

Actual Behavior

Unable to build bundle with serverless-webpack

Steps to Reproduce the Problem

$ sls create -t aws-nodejs-ecma-script -p dd-webpack
$ cd dd-webpack/
$ yarn add datadog-lambda-js
$ sed -i '1s/^/const { datadog } = require("datadog-lambda-js");\n/' first.js # Add library to beginning of file
$ sls package

Specifications

• Datadog Lambda Layer version: v0.5
• Node version: 8.16.0

Stacktrace

Serverless: Bundling with Webpack...
Time: 2821ms
Built at: 2019-10-24 18:12:00
  Asset      Size  Chunks             Chunk Names
first.js  7.84 MiB   first  [emitted]  first
second.js  4.42 KiB  second  [emitted]  second
Entrypoint first = first.js
Entrypoint second = second.js
[./first.js] 374 bytes {first} [built]
[./node_modules/aws-sdk/lib/aws.js] 159 bytes {first} [optional] [built]
[./node_modules/datadog-lambda-js/dist/index.js] 9.99 KiB {first} [built]
[./node_modules/datadog-lambda-js/dist/metrics/api.js] 2.51 KiB {first} [built]
[./node_modules/datadog-lambda-js/dist/metrics/index.js] 297 bytes {first} [built]
[./node_modules/datadog-lambda-js/dist/metrics/kms-service.js] 4.33 KiB {first} [built]
[./node_modules/datadog-lambda-js/dist/metrics/listener.js] 8.94 KiB {first} [built]
[./node_modules/datadog-lambda-js/dist/metrics/model.js] 2.38 KiB {first} [built]
[./node_modules/datadog-lambda-js/dist/trace/index.js] 200 bytes {first} [built]
[./node_modules/datadog-lambda-js/dist/trace/listener.js] 4.83 KiB {first} [built]
[./node_modules/datadog-lambda-js/dist/utils/handler.js] 5.35 KiB {first} [built]
[./node_modules/datadog-lambda-js/dist/utils/index.js] 420 bytes {first} [built]
[./node_modules/datadog-lambda-js/dist/utils/log.js] 1.63 KiB {first} [built]
[./node_modules/datadog-lambda-js/dist/utils/timer.js] 2.52 KiB {first} [built]
[./second.js] 411 bytes {second} [built]
  + 1153 hidden modules

WARNING in ./node_modules/dd-trace/packages/dd-trace/src/platform/node/loader.js 143:11-31
Critical dependency: the request of a dependency is an expression
@ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
@ ./node_modules/dd-trace/packages/dd-trace/index.js
@ ./node_modules/dd-trace/index.js
@ ./node_modules/datadog-lambda-js/dist/trace/listener.js
@ ./node_modules/datadog-lambda-js/dist/trace/index.js
@ ./node_modules/datadog-lambda-js/dist/index.js
@ ./first.js

WARNING in ./node_modules/colors/lib/colors.js 127:29-43
Critical dependency: the request of a dependency is an expression
@ ./node_modules/colors/safe.js
@ ./node_modules/winston/lib/winston/config.js
@ ./node_modules/winston/lib/winston.js
@ ./node_modules/aws-xray-sdk-core/lib/logger.js
@ ./node_modules/aws-xray-sdk-core/lib/context_utils.js
@ ./node_modules/aws-xray-sdk-core/lib/aws-xray.js
@ ./node_modules/aws-xray-sdk-core/lib/index.js
@ ./node_modules/datadog-lambda-js/dist/trace/context.js
@ ./node_modules/datadog-lambda-js/dist/trace/listener.js
@ ./node_modules/datadog-lambda-js/dist/trace/index.js
@ ./node_modules/datadog-lambda-js/dist/index.js
@ ./first.js

ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/printer' in '/private/tmp/dd-webpack/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 10:18-53
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
@ ./node_modules/dd-trace/packages/dd-trace/index.js
@ ./node_modules/dd-trace/index.js
@ ./node_modules/datadog-lambda-js/dist/trace/listener.js
@ ./node_modules/datadog-lambda-js/dist/trace/index.js
@ ./node_modules/datadog-lambda-js/dist/index.js
@ ./first.js

ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/language/visitor' in '/private/tmp/dd-webpack/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 9:18-53
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
@ ./node_modules/dd-trace/packages/dd-trace/index.js
@ ./node_modules/dd-trace/index.js
@ ./node_modules/datadog-lambda-js/dist/trace/listener.js
@ ./node_modules/datadog-lambda-js/dist/trace/index.js
@ ./node_modules/datadog-lambda-js/dist/index.js
@ ./first.js

ERROR in ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js
Module not found: Error: Can't resolve 'graphql/utilities' in '/private/tmp/dd-webpack/node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools'
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/transforms.js 11:20-48
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/signature.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/tools/index.js
@ ./node_modules/dd-trace/packages/datadog-plugin-graphql/src/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/plugins/index.js
@ ./node_modules/dd-trace/packages/dd-trace/src/platform/node/index.js
@ ./node_modules/dd-trace/packages/dd-trace/index.js
@ ./node_modules/dd-trace/index.js
@ ./node_modules/datadog-lambda-js/dist/trace/listener.js
@ ./node_modules/datadog-lambda-js/dist/trace/index.js
@ ./node_modules/datadog-lambda-js/dist/index.js
@ ./first.js

Error --------------------------------------------------

Error: Webpack compilation error, see above
    at _.forEach.compileStats (/private/tmp/dd-webpack/node_modules/serverless-webpack/lib/compile.js:38:19)
    at arrayEach (/private/tmp/dd-webpack/node_modules/lodash/lodash.js:516:11)
    at Function.forEach (/private/tmp/dd-webpack/node_modules/lodash/lodash.js:9342:14)
    at BbPromise.fromCallback.then.stats (/private/tmp/dd-webpack/node_modules/serverless-webpack/lib/compile.js:31:11)
    at tryCatcher (/private/tmp/dd-webpack/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/private/tmp/dd-webpack/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/private/tmp/dd-webpack/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/private/tmp/dd-webpack/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/private/tmp/dd-webpack/node_modules/bluebird/js/release/promise.js:729:18)
    at Promise._fulfill (/private/tmp/dd-webpack/node_modules/bluebird/js/release/promise.js:673:18)
    at /private/tmp/dd-webpack/node_modules/bluebird/js/release/nodeback.js:42:21
    at finalCallback (/private/tmp/dd-webpack/node_modules/webpack/lib/Compiler.js:257:39)
    at hooks.done.callAsync.err (/private/tmp/dd-webpack/node_modules/webpack/lib/Compiler.js:306:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/private/tmp/dd-webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook (/private/tmp/dd-webpack/node_modules/tapable/lib/Hook.js:154:20)
    at emitRecords.err (/private/tmp/dd-webpack/node_modules/webpack/lib/Compiler.js:304:22)
    at Compiler.emitRecords (/private/tmp/dd-webpack/node_modules/webpack/lib/Compiler.js:499:39)
    at emitAssets.err (/private/tmp/dd-webpack/node_modules/webpack/lib/Compiler.js:298:10)
    at hooks.afterEmit.callAsync.err (/private/tmp/dd-webpack/node_modules/webpack/lib/Compiler.js:485:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/private/tmp/dd-webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook (/private/tmp/dd-webpack/node_modules/tapable/lib/Hook.js:154:20)
    at asyncLib.forEachLimit.err (/private/tmp/dd-webpack/node_modules/webpack/lib/Compiler.js:482:27)
    at /private/tmp/dd-webpack/node_modules/neo-async/async.js:2818:7
    at done (/private/tmp/dd-webpack/node_modules/neo-async/async.js:3522:9)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/private/tmp/dd-webpack/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at outputFileSystem.writeFile.err (/private/tmp/dd-webpack/node_modules/webpack/lib/Compiler.js:464:33)
    at /Users/simon.males/.npm-packages/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:111:16
    at /private/tmp/dd-webpack/node_modules/graceful-fs/graceful-fs.js:57:14
    at /Users/simon.males/.npm-packages/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:45:10
    at FSReqWrap.oncomplete (fs.js:135:15)

   For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Get Support --------------------------------------------
   Docs:          docs.serverless.com
   Bugs:          github.com/serverless/serverless/issues
   Issues:        forum.serverless.com

Your Environment Information ---------------------------
   Operating System:          darwin
   Node Version:              8.16.0
   Framework Version:         1.55.1
   Plugin Version:            3.2.0
   SDK Version:               2.1.2
   Components Core Version:   1.1.2
   Components CLI Version:    1.4.0  ```

Ability to override http request options

Expected Behavior

Can somehow configure the underlying http(s) client to override values https://nodejs.org/api/http.html#http_http_request_options_callback

Specifically looking to set the timeout values to something lower than the default 120 seconds.
We have had cases, under high load, where the datadog client library would timeout and cause our lambdas (that were configured for less than 120 seconds execution time) to fail.

Actual Behavior

Currently there is no way to set these values

Version 2.23.0 Fails to Build

Expected Behavior

I expected 2.23.0 to build with Parcel just like 2.22.0 did.

Actual Behavior

I get a build failure when I try to build with 2.23.0.

🚨  /private/tmp/test-datadog/node_modules/cls-hooked/context.js:7:28: Cannot resolve dependency 'async_hooks'
   5 | const assert = require('assert');
   6 | const wrapEmitter = require('emitter-listener');
>  7 | const async_hooks = require('async_hooks');
     |                            ^
   8 |
   9 | const CONTEXTS_SYMBOL = 'cls@contexts';
  10 | const ERROR_SYMBOL = 'error@context';

Steps to Reproduce the Problem

Follow the steps in this sample repo: https://github.com/blimmer/datadog-lambda-layer-js-issue (see README)

Specifications

• Datadog Lambda Layer version: 2.23.0
• Node version: 12.16.3 (12 LTS)

Stacktrace

🚨  /private/tmp/test-datadog/node_modules/cls-hooked/context.js:7:28: Cannot resolve dependency 'async_hooks'
 5 | const assert = require('assert');
 6 | const wrapEmitter = require('emitter-listener');
>  7 | const async_hooks = require('async_hooks');
   |                            ^
 8 |
 9 | const CONTEXTS_SYMBOL = 'cls@contexts';
10 | const ERROR_SYMBOL = 'error@context';

Ability to disable built-in tags for sendDistributionMetric function.

Expected Behavior

When I'm using NPM package datadog-lambda-js and I'm sending metrics using function sendDistributionMetric(name: string, value: number, ...tags: string[]): void; I end up receiving additional custom metrics with built-in tags, for example dd_lambda_layer:datadog-nodev12.14.1.

Since any unique combination of metric name and tag name is billed as separate custom metric, I would like to disable this behaviour, since I'm using datadog-lambda-js as DataDog API client for custom metrics.

Actual Behavior

Calling function sendDistributionMetric(name: string, value: number, ...tags: string[]): void; from datadog-lambda-js NPM package adds additional, built-in tags, for example dd_lambda_layer:datadog-nodev12.14.1.

Steps to Reproduce the Problem

1. Create Node.js Lambda and add "datadog-lambda-js": "2.17.0" as dependency.
2. Use sendDistributionMetric function.
3. In DataDog UI, metrics explorer you will see some additional tags, that you haven't specified in the code.

Specifications

• Datadog Lambda Layer version: None, "datadog-lambda-js": "2.17.0"
• Node version: 12.x

Sending metrics from asynchronous handler that encounters an error?

I have a function fed by SQS that looks similar to:

module.exports.handler = datadog(lambdaHandler, {
  apiKey: process.env.DATDOG_API_KEY,
  logForwarding: true,
  injectLogContext: true
});

async function lambdaHandler(event) {
  try {
    const payload = JSON.parse(event.Records[0].body)
    // do some stuff
  } catch (error) {
    const error = Error('failed to handle incoming record');
    error.cause = cause;
    sendMetric('error.handle-record');
    throw error;
  }
}

function sendMetric(name, value = 1) {
  sendDistributionMetricWithDate(`foo.${name}`, value, new Date(), [
    'function:foo',
    'env:dev'
  ]);
}

The idea being I will increment a DataDog metric before I tell the Lambda system to re-queue the failed message. But I never see the metric in my DD dashboard. I believe the function is being killed before the sendDistributionMetricWithDate has a chance to complete its work.

I've been looking through the source code, but I'm not a TypeScript person so I'm not clear: can sendDistributionMetricWithDate be awaited or have a callback passed to it to indicate it is complete?

I'm currently using "datadog-lambda-js": "~3.43.0".

SERVERLESS SNS HANDLER

import {Context, SNSHandler, SNSEvent} from 'aws-lambda';
import {datadog, sendDistributionMetric} from "datadog-lambda-js";

const handler: SNSHandler = async (event : SNSEvent, context : Context) : Promise < void > => {
await sendMetric(1, "hello world");
}

async function sendMetric(tenantId : any, message: any) {
sendDistributionMetric(
"impoart_failure.data_value", // Metric name
1, // Metric value
environment: ${process.env.ENVIORNMENT}: ${tenantId},
message: ${message} // Associated tags
);
}

const dd = datadog(handler);
export {
dd
};

PLEASE PROVIDE EXAMPLE FOR SNSHANDLER CASE,ABOVE CODE DOES NOT WORK

Cannot find layer Datadog-Node10-x:29

Expected Behavior

Should be able to get layer information for Datadog-Node10-x:29
When I deploy with the latest version of serverless-plugin-datadog it tries to add layer Datadog-Node10-x:29 to my service.
However, it appears that the layer does not exist

Actual Behavior

➜ aws lambda get-layer-version-by-arn --arn arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Node10-x:29

An error occurred (AccessDeniedException) when calling the GetLayerVersionByArn operation: User: arn:aws:sts::699536110035:assumed-role/DeveloperAdmin/jonathan.schmidt is not authorized to perform: lambda:GetLayerVersion on resource: arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Node10-x:29

Steps to Reproduce the Problem

1. Use AWS CLI to get layer information

aws lambda get-layer-version-by-arn --arn arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Node10-x:29

I can get version 28:

➜ aws lambda get-layer-version-by-arn --arn arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Node10-x:28
{
    "Content": {
        "Location": "https://awslambda-us-west-2-layers.s3.us-west-2.amazonaws.com/snapshots/464622532012/Datadog-Node10-x-ea3343cc-7dd2-489a-86fe-0b56d27dd134?versionId=jxsp.no8EhuCkC72aaLzEoJheHfTPmyF&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEG8aCXVzLXdlc3QtMiJGMEQCID2PNIyW0Z4WvrsC2Jk3cs9BvaaId1gJDXVWXrel%2BE3XAiAOV1i%2B0uLYhbr3Bp%2BQfGONntVhasOEXQOBlZclVA6%2BVCq0AwhnEAIaDDUwMjI5NzA3NjE2MyIMLn5NuHhttlczDtQFKpED2n67SmMOZPdSE9GqYNwJi6ZgxZUabwVgs%2BTCUHm2t4uUiRQ1i25vZsl6%2B%2BpyjYM4QIV1HdFyiaD828%2B3JNlexbKmF85FNNNve8y2CAxco98B6Bw84J8zGfpZ%2BmZP6dYoxXjdnaJoVcXFjZO4naBQ%2B5zmIxZsDUZyemcztjShE3y4FEcXrztv%2FNmI8Okl1FMGuAGOnGjaL1ozImzcVfEyjvxgqawMk4MKfL69suRJkZgkqZl5Vdj%2BgcG8BiTboje4HfX%2FZvVMFV4SWC7g84C3s%2BXOGKJOSD2sVxaDzaWP5Lfu11iBbHyrB3%2FeN0GJoNBgT6%2B8c9rDpmWy8xbC%2FnyuuXW8iF6Br2rYZQ8VrYcaFErqQDVdBTQdjF12TgbasLx3Oh6TW%2BZaVRLJaCxh2IYS5gc71zRbF7RhhttJLTu%2FipOGWf0l%2Bdidd6fxtHtGt2nBpkSuQgvBh66y5BYTVEvigKuHwPmaaOSIk6MeEHGA2FOb2RLdHlcTQl0WGe%2F1NTH0dzDIkq%2Bhgaw7ypQsXGprb14w4e61%2BgU67AEeD89L43PF462X1H751oxVampwEDLjdJYgbjtMniCUWf6q3ilZrfYo3ZNEcarI2ZsVL%2FhfPJwdEU5AmhBjR1yZwCsysl1FazfFQYTZ1UhyJzCR4uIgj%2F%2FI846Xy3vBDQ2XHHC9uxgCfYYAGUC6sxkrvMV3uNeF7U7yC4s15oA%2FH5%2BSRHMi1j005XKPcFpL1VwBAUmu98jwKnI4v0r0%2FVq3NbF%2B4avo9uJHX7UU1Hrs4nH1uPGTFHE9iZjZTtx1Vh%2BL1pl388K8LguGFTjC%2Baa2YVvedWzujGwdojVnGI042pxi34rN6Q0k9SVo3w%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200831T225710Z&X-Amz-SignedHeaders=host&X-Amz-Expires=600&X-Amz-Credential=ASIAXJ4Z5EHBVVRXCR2S%2F20200831%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=6b8f4d9884cc0bc653e61177c29f04f1e52d36d7268bb30b46f7c3a37fe072b0",
        "CodeSha256": "xOrEANx5gCGFvQXiET9JHeQ8OMSSusyFU28koEQv8Vs=",
        "CodeSize": 5105432
    },
    "LayerArn": "arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Node10-x",
    "LayerVersionArn": "arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Node10-x:28",
    "Description": "Datadog Lambda Layer for Node",
    "CreatedDate": "2020-08-20T18:05:35.816+0000",
    "Version": 28,
    "CompatibleRuntimes": [
        "nodejs10.x"
    ]
}

Question: AWS SDK Lambda invoke tracing

Hey

In the distributed tracing docs it says

Tracing synchronous Lambda invokes with the AWS SDK (coming soon for Node.js).
Tracing asynchronous Lambda invocations through AWS SQS (coming soon for Node.js).

Is there a rough timeline for when this will be ready?

Lambda reports error "failed to flush metrics" with no more detail.

Expected Behavior

I have a lamda function that makes and async http call. I expect the http response data to be sent to data dog as metric value.

const https = require("https");
const { datadog, sendDistributionMetric } = require("datadog-lambda-js");

const domain = "dev.domain";
const url = `https://${domain}/getMetric/`;

function send(desc, value) {
  console.log(`${desc}: ${value}`);
  sendDistributionMetric(desc, value);
}

const prefix = `lambda.${domain.replace(".", "_")}`;

function execHandler(event) {
  const promise = new Promise(function (resolve, reject) {
    https
      .get(url, (res) => {
        var body = "";
        res.on("data", function (chunk) {
          body += chunk;
        });
        res.on("end", function () {
          const result = JSON.parse(body);

          send(`${prefix}.status_code`, parseInt(res.statusCode));
          send(`${prefix}.value`, parseInt(result.val));
          resolve(res.statusCode);
        });
      })
      .on("error", (e) => {
        send(`${prefix}.request_failed`, 1);
        reject(Error(e));
      });
  });
  return promise;
}

module.exports.execHandler = datadog(execHandler);

exports.handler = datadog(execHandler);

Actual Behavior

Lambda runs but ends with failed to flush

Steps to Reproduce the Problem

1. Add script above to lamda
2. Set lambda layer to "arn:aws:lambda:eu-central-1:464622532012:layer:Datadog-Node12-x:43"
3. Run

Specifications

• Datadog Lambda Layer version: 43
• Node version: Node12

Stacktrace

2021-01-25T12:34:47.273Z	044bb758-93eb-4721-8016-44462c29ec07	INFO	[dd.trace_id=702867017315363974 dd.span_id=7792049542437625856] lambda.dev_domain.status_code: 200
2021-01-25T12:34:47.273Z	044bb758-93eb-4721-8016-44462c29ec07	INFO	[dd.trace_id=702867017315363974 dd.span_id=7792049542437625856] lambda.dev_domain.value: 47
2021-01-25T12:34:47.656Z	044bb758-93eb-4721-8016-44462c29ec07	ERROR	[dd.trace_id=702867017315363974 dd.span_id=7792049542437625856] {"innerError":{},"status":"error","message":"datadog:failed to flush metrics"}

patch-http not supporting "new" node URL object, breaking all outgoing calls

Summary

Deviating from the template to give you a summary that'll hopefully quickly communicate the problem:
I think the node HTTP module patching in this project isn't handling a common call pattern. Looking at the code, the patch-http file seems to only support what the node docs call the legacy URL object, which is now deprecated. Things fail if an http.request() call is made using the new node URL object.

Expected Behavior

Patching the node HTTP module doesn't interfere with the rest of the application. HTTP requests work as expected.

Actual Behavior

All HTTP requests are being sent to localhost because the URL portion isn't interpreted correctly.

Steps to Reproduce the Problem

I think this test case should help you reproduce it if you add it to your src/trace/patch-http.spec.ts file:

  it("injects tracing headers when using the new WHATWG URL object", () => {
    nock("http://www.example.com").get("/").reply(200, {});
    patchHttp(contextService);
    // const url = parse("http://www.example.com");
    const url = new URL("http://www.example.com");
    const req = http.request(url);
    expectHeaders(req);
  });

Specifications

• Datadog Lambda Layer version: My app failed with v3.29.0, but I did try the unit test above with your latest master v3.41.0.
• Node version: 14.3.0

Stacktrace

Not relevant

Detailed description of the problem

I have a lambda that is wrapped with datadog-lambda-js. The lambda's business logic needs to send out some HTTP requests. I'm using the GOT library for that: got

With the wrapping, all http requests are being sent to 127.0.0.1:443! Things work correctly if I remove datadog-lambda-js.

After much code reading, I think I've identified the problem. Your code from patch-http.ts:

function normalizeArgs(
  arg1: string | URL | http.RequestOptions,
  arg2?: RequestCallback | http.RequestOptions,
  arg3?: RequestCallback,
) {
  let options: http.RequestOptions = typeof arg1 === "string" ? parse(arg1) : { ...arg1 };

The got library is calling http.request with 2 args where arg1 is an instance of the "new" URL object in node. The object is not a simple JS object that you can use a spread operator on, as is being done in the ternary statement above.

The code above looks like it only works with the now deprecated legacy URL object. The legacy one was returning simple objects.