datadog / datadog-lambda-rb

The Datadog AWS Lambda Layer for Ruby

datadog-lambda-rb

Datadog Lambda Library for Ruby (2.7, and 3.2) enables enhanced Lambda metrics, distributed tracing, and custom metric submission from AWS Lambda functions.

Installation

Follow the installation instructions, and view your function's enhanced metrics, traces and logs in Datadog.

Configuration

Follow the configuration instructions to tag your telemetry, collect telemetry from non-Lambda resources, and more.

For additional configuraton options for distributed tracing, see the Datadog trace client readme

Opening Issues

If you encounter a bug with this package, we want to hear about it. Before opening a new issue, search the existing issues to avoid duplicates.

When opening an issue, include the Datadog Lambda Layer version, Ruby version, and stack trace if available. In addition, include the steps to reproduce when appropriate.

You can also open an issue for a feature request.

Contributing

If you find an issue with this package and have a fix, please feel free to open a pull request following the procedures.

Community

For product feedback and questions, join the #serverless channel in the Datadog community on Slack.

License

Unless explicitly stated otherwise all files in this repository are licensed under the Apache License Version 2.0.

This product includes software developed at Datadog (https://www.datadoghq.com/). Copyright 2019 Datadog, Inc.

datadog-lambda-rb's Issues

Update datadog-lambda to work with ddrace gem versions >= 1.0

Expected Behavior

The latest datadog-lambda gem works with the latest ddtrace gem.

Actual Behavior

The latest datadog-lambda gem doesn't work with the latest ddtrace gem.

Steps to Reproduce the Problem

1. Add the latest datadog-lambda and ddtrace gems.
2. Add an initializer with the Datadog::Lambda.configure_apm block.
3. Boot app.
4. App fails due to datadog-lambda's dependency on SyncWriter which is not supported in ddtrace versions >= 1.0.

See: https://github.com/DataDog/dd-trace-rb/blob/master/docs/UpgradeGuide.md#from-0x-to-10

This dependency can be found here:

datadog-lambda-rb/lib/datadog/lambda.rb

Line 31 in dc40377

require 'ddtrace/sync_writer'

Specifications

• Datadog Lambda Layer version: 1.14.0
• ddtrace: 1.1.0
• Ruby version: 3.0

Stacktrace

/usr/local/bundle/gems/bootsnap-1.9.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require': cannot load such file -- ddtrace/sync_writer (LoadError)

Ruby 3.1 support

Overview

The AWS lambda stack only supports Ruby 2.7 when running in the "Amazon Linux" environment. However, AWS lambda also supports running Docker Images.

These Docker images can be any version of Ruby - including version 3.1.

This gem seems to only support Ruby 2.7 - while Ruby 3.0 was released in December 2020, and Ruby 3.1 was released in December 2021.

For those of us choosing to use a recent docker-based image (instead of the S3 file based lambda image where only 2.7 is supported) we are unable to use this gem.

Attempting to use this gem with Ruby 3.1 gives the following error, due to the way that later versions of Ruby handle keyword arguments.

Expected Behavior

This gem would work with Ruby 2.7 and later versions of Ruby such as 3.1

Actual Behavior

The gem doesn't support tracing with Ruby 3.1.

Steps to Reproduce the Problem

1. Install Ruby 3.1
2. Try and use the Datadog::Lambda.wrap method
3. You will receive a message saying "wrong number of arguments (given 2, expected 1) (ArgumentError)"

Specifications

• Datadog Lambda Layer version: Not relevant - using a docker image
• Ruby version: ruby:3.1.3-slim docker image

Stacktrace

/Users/oskar/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/ddtrace-1.7.0/lib/datadog/tracing.rb:17:in `trace': wrong number of arguments (given 2, expected 1) (ArgumentError)
  from /Users/oskar/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/datadog-lambda-2.16.0/lib/datadog/lambda/trace/ddtrace.rb:36:in `wrap_datadog'
  from /Users/oskar/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/datadog-lambda-2.16.0/lib/datadog/lambda/trace/listener.rb:51:in `on_wrap'
  from /Users/oskar/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/datadog-lambda-2.16.0/lib/datadog/lambda.rb:57:in `wrap'
  from (irb):39:in `block in <top (required)>'

A monkey-patch such as the following fixes this issue, but there may be other similar issues. The options parameter needs to be prefixed with **

module Datadog
  module Trace
    class <<self
      def wrap_datadog(options = {}, &block)
        unless Datadog::Tracing.enabled?
          Datadog::Utils.logger.debug 'dd-trace unavailable'
          return block.call
        end

        Datadog::Tracing.trace('aws.lambda', **options) do |_span|
          block.call
        end
      end
    end
  end
end

DD_SERVICE env variable isn't respected for ddtrace

Expected Behavior

DD_SERVICE environment variable should take precedence over AWS_LAMBDA_FUNCTION_NAME for APM service name based on ddtrace documentation.

It already works like this in datadog-lambda-layer-js

Actual Behavior

DD_SERVICE environment variable is ignored. I assume it's because lambda layer hard codes it to function name.

I'm wondering if it would be straightforward to have it configured during ddtrace configuration here instead of hardcoding.

Steps to Reproduce the Problem

1. Create ruby lambda function with name dd-test with environment variable DD_SERVICE=dd-new-name
2. Instrument with ddtrace and datadog-lambda gems
3. APM will show service named dd-test instead of dd-new-name

Specifications

• Datadog Lambda gem version: 0.7.0
• ddtrace gem version: 0.36.0
• Ruby version: 2.7

DDTrace Connection Refused

Expected Behavior

The lambda runs without datadog tracer errors

Actual Behavior

The lambda has a connection error error, see stack trace.

Steps to Reproduce the Problem

1.Create a lambda

1. Give minimal code for logging with trace ids

require 'logger'
require 'datadog/lambda'

def lambda_handler(event:, context:)
    Datadog::Lambda.wrap(event, context) do
        time = Time.new
        logger = Logger.new($stdout, formatter: proc {|severity, datetime, progname, msg|
            "#{time.strftime("%Y-%m-%dT%H:%M:%S.%3NZ")} #{context.aws_request_id} [#{severity}] [dd.trace_id=#{Datadog.tracer.active_correlation.trace_id} dd.span_id=#{Datadog.tracer.active_correlation.span_id}] #{msg}"})
        logger.info(event.to_json)
    end
end

1. Lambda Config:

{
    "FunctionName": "EventBridgeLoggerDev",
    "FunctionArn": "arn:aws:lambda:us-west-2:127579856528:function:EventBridgeLoggerDev",
    "Runtime": "ruby2.5",
    "Role": "arn:aws:iam::127579856528:role/AWSLambdaBasicExecutionRole",
    "Handler": "main.lambda_handler",
    "CodeSize": 649,
    "Description": "Logs all events for an EventBridge to CloudWatch Logs",
    "Timeout": 10,
    "MemorySize": 128,
    "LastModified": "2020-10-29T20:34:28.917+0000",
    "CodeSha256": "Z7Lyamqk7gF+RgJXWFb5j+7AuboztcVR0EhVL115PhE=",
    "Version": "$LATEST",
    "Environment": {
        "Variables": {
            "DD_SERVICE": "EventBridgeLogger",
            "DD_LOGS_INJECTION": "true",
            "DD_ENV": "dev",
            "DD_LOG_LEVEL": "debug"
        }
    },
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "RevisionId": "fabd3c5b-e3e5-4aa4-ab0b-ec32c4a9c822",
    "Layers": [
        {
            "Arn": "arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Ruby2-5:11",
            "CodeSize": 6891257
        }
    ],
    "State": "Active",
    "LastUpdateStatus": "Successful"
}

Specifications

• Datadog Lambda Layer version: 11 (arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Ruby2-5:11)

• Ruby version: 2.5

Stacktrace

Response:
true

Request ID:
"243aa928-a647-486e-9054-ae0f7bd815ba"

Function logs:
-ruby25","functionname:EventBridgeLoggerDev","region:us-west-2","account_id:127579856528","memorysize:128","cold_start:true","runtime:Ruby 2.5.8","resource:EventBridgeLoggerDev","datadog_lambda:1.11.0","dd_trace:0.41.0"],"v":1}
2020-10-29T20:34:49.475Z 243aa928-a647-486e-9054-ae0f7bd815ba [INFO] [dd.trace_id=3637116462645295449 dd.span_id=1572314633988182015] {"key1":"value1","key2":"value2","key3":"value3"}D, [2020-10-29T20:34:49.495486 #9] DEBUG -- ddtrace: [ddtrace] (/opt/ruby/gems/2.5.0/gems/datadog-lambda-1.11.0/lib/datadog/lambda/trace/patch_http.rb:41:in `rescue in request') couldn't add tracing context  to request odd number of arguments for Hash:
/opt/ruby/gems/2.5.0/gems/datadog-lambda-1.11.0/lib/datadog/lambda/trace/xray.rb:71:in `[]'
/opt/ruby/gems/2.5.0/gems/datadog-lambda-1.11.0/lib/datadog/lambda/trace/xray.rb:71:in `current_trace_context'
/opt/ruby/gems/2.5.0/gems/datadog-lambda-1.11.0/lib/datadog/lambda/trace/patch_http.rb:34:in `request'
/var/lang/lib/ruby/2.5.0/net/http.rb:1481:in `send_entity'
/var/lang/lib/ruby/2.5.0/net/http.rb:1269:in `post'
/var/lang/lib/ruby/2.5.0/net/http.rb:506:in `block in post'
/var/lang/lib/ruby/2.5.0/net/http.rb:910:in `start'
/var/lang/lib/ruby/2.5.0/net/http.rb:609:in `start'
/var/lang/lib/ruby/2.5.0/net/http.rb:504:in `post'
/var/runtime/lib/lambda_server.rb:47:in `send_response'
/var/runtime/lib/runtime.rb:48:in `<main>'
E, [2020-10-29T20:34:49.515616 #9] ERROR -- ddtrace: [ddtrace] (/opt/ruby/gems/2.5.0/gems/ddtrace-0.41.0/lib/ddtrace/transport/http/client.rb:35:in `rescue in send_request') Internal error during HTTP transport request. Cause: Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126) Location: /var/lang/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect'
I, [2020-10-29T20:34:49.516104 #9]  INFO -- ddtrace: [ddtrace] DATADOG TRACER CONFIGURATION - {"date":"2020-10-29T20:34:49+00:00","os_name":"x86_64-pc-linux-gnu","version":"0.41.0","lang":"ruby","lang_version":"2.5.8","env":"dev","enabled":true,"service":"EventBridgeLogger","agent_url":"http://127.0.0.1:8126?timeout=1","agent_error":"Datadog::Transport::InternalErrorResponse ok?: unsupported?:, not_found?:, client_error?:, server_error?:, internal_error?:true, payload:, error_type:Errno::ECONNREFUSED error:Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for \"127.0.0.1\" port 8126)","debug":false,"analytics_enabled":false,"tags":"env:dev","runtime_metrics_enabled":false,"vm":"ruby-2.5.8","partial_flushing_enabled":false,"priority_sampling_enabled":false,"health_metrics_enabled":false}
W, [2020-10-29T20:34:49.516159 #9]  WARN -- ddtrace: [ddtrace] DATADOG TRACER DIAGNOSTIC - Agent Error: Datadog::Transport::InternalErrorResponse ok?: unsupported?:, not_found?:, client_error?:, server_error?:, internal_error?:true, payload:, error_type:Errno::ECONNREFUSED error:Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126)
D, [2020-10-29T20:34:49.535445 #9] DEBUG -- ddtrace: [ddtrace] (/opt/ruby/gems/2.5.0/gems/datadog-lambda-1.11.0/lib/datadog/lambda/trace/patch_http.rb:41:in `rescue in request') couldn't add tracing context  to request odd number of arguments for Hash:
/opt/ruby/gems/2.5.0/gems/datadog-lambda-1.11.0/lib/datadog/lambda/trace/xray.rb:71:in `[]'
/opt/ruby/gems/2.5.0/gems/datadog-lambda-1.11.0/lib/datadog/lambda/trace/xray.rb:71:in `current_trace_context'
/opt/ruby/gems/2.5.0/gems/datadog-lambda-1.11.0/lib/datadog/lambda/trace/patch_http.rb:34:in `request'
/var/lang/lib/ruby/2.5.0/net/http.rb:1216:in `get'
/var/runtime/lib/lambda_server.rb:21:in `block in next_invocation'
/var/lang/lib/ruby/2.5.0/net/http.rb:910:in `start'
/var/runtime/lib/lambda_server.rb:20:in `next_invocation'
/var/runtime/lib/runtime.rb:30:in `<main>'
END RequestId: 243aa928-a647-486e-9054-ae0f7bd815ba
REPORT RequestId: 243aa928-a647-486e-9054-ae0f7bd815ba	Duration: 73.74 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 57 MB	Init Duration: 583.78 ms	

Missing invoked arn in Sam throwing error

Expected Behavior

I am trying to test my functions that use this layer locally using AWS Sam.

Actual Behavior

When Sam executes the function I get an error from this line:

datadog-lambda-rb/lib/datadog/lambda/trace/listener.rb

Line 59 in c15f0f0

function_arn = request_context.invoked_function_arn.downcase

I found that context.invoked_function_arn is set to nil.

Steps to Reproduce the Problem

1. Create a new Sam function with sam init. I used the Ruby 2.7 hello world example function.
2. Wrap the code in the Datadog Lambda layer
3. Run sam local invoke "HelloWorldFunction" --no-event

Specifications

• Datadog Lambda Layer version: 1.12 (other version also throw this error)
• Ruby version: 2.7 or 2.5
• Sam version: 1.18.0 (also tried with 1.15.0)

Stacktrace

{
"errorMessage": "undefined method `downcase' for nil:NilClass",
"errorType": "Function<NoMethodError>",
"stackTrace": [
  "/opt/ruby/gems/2.7.0/gems/datadog-lambda-1.12.0/lib/datadog/lambda/trace/listener.rb:59:in `get_option_tags'",
  "/opt/ruby/gems/2.7.0/gems/datadog-lambda-1.12.0/lib/datadog/lambda/trace/listener.rb:40:in `on_wrap'",
  "/opt/ruby/gems/2.7.0/gems/datadog-lambda-1.12.0/lib/datadog/lambda.rb:57:in `wrap'",
  "/var/task/app.rb:19:in `lambda_handler'"
]
}

I tried several different local execution environments on different hosts and as far as I can tell, context.invoked_function_arn isn't ever set when running via Sam. I don't believe this was true in the past, and I can't find any specific change that is causing this.

Allow custom request extraction method for SNS/SQS message attributes

Expected Behavior

Asynchronous Lambda invocations triggers by SNS/SQS events could be correlated back to the original publisher trace.

This seems to be supported by the Python and Node.js wrappers: https://docs.datadoghq.com/serverless/distributed_tracing/serverless_trace_propagation?tab=python

Actual Behavior

Asynchrounous Lambda invocations always show up as standalone traces in APM.

In our infrastructure we do pass Datadog headers as SNS/SQS message attributes and could reify the Tracer context with this information if there was a way to extract it differently that today's method which only looks for synchronous API calls with headers.

Steps to Reproduce the Problem

1. Send an SNS event to a Lambda subscription
2. Check APM trace without distributed tracing

Specifications

• Datadog Lambda Layer version:
• Ruby version: 2.7.1

Stacktrace

Paste here

Unable to use layer in Lambda function

Expected Behavior

The arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Ruby:2 layer can be used in a Lambda function.

Actual Behavior

An error message is presented:

You are not authorized to perform: lambda:GetLayerVersion.

Steps to Reproduce the Problem

1. Create a lambda function
2. Add arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Ruby:2 layer
3. Press "Save"

Specifications

• Datadog Lambda Layer version: arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Ruby:2
• Ruby version: 2.5

Stacktrace

You are not authorized to perform: lambda:GetLayerVersion.