sumologic / sumologic-aws-lambda

A collection of lambda functions to collect data from Cloudwatch, Kinesis, VPC Flow logs, S3, security-hub and AWS Inspector

Sumo Logic AWS Lambda Functions

What does AWS Lambda do?

AWS Lambda is a compute service that allows users to run code, in response to events, without having to provision and manage servers. A Lambda Function can be triggered automatically from other Amazon services, or from a web or mobile application. For more information, please visit the AWS Lambda site.

What do Sumo Logic Lambda Functions do?

Sumo Logic Lambda Functions are designed to collect and process data from a variety of sources and pass it onto the Sumo Logic platform. Here, the data can be stored, aggregated, searched, and visualized for a variety of insightful use cases.

What are the different Sumo Logic Lambda Functions available?

We put the Lambda functions to read from a particular AWS service (e.g CloudWatch Logs and S3) under each specific folder. Each folder may then have its own instructions to setup the functions.

Collection Solutions

FunctionName	Description	Collection Use Cases	Setup Documentation
SumoLogic Lambda Function for AWS CloudWatch Logs With Dead Letter Queue Support	This project comes with Cloudformation template and two lambda functions which sends CloudWatch logs to Sumo Logic HTTP source endpoint.The first function(invoked by CloudWatch) is configured with DLQ and the second function(invoked periodically by CloudWatch Events) reads from DLQ.	AWS Lambda ULM App	Docs
SumoLogic Function for AWS CloudWatch Events	This function is invoked by AWS CloudWatch events in response to state change in your AWS resources which matches a event target definition. The event payload received is then forwarded to Sumo Logic HTTP source endpoint.	AWS GuardDuty App	Docs
SumoLogic Function for Amazon Inspector	This function subscribes to a SNS topic where Amazon Inspector publishes its findings.It receives the message payload as an input parameter, transforms it and sends it to Sumo Logic HTTP source endpoint	Amazon Inspector	Docs
AWS Kinesis Firehose for Metrics	This project sets up AWS resources to Send AWS CloudWatch metrics using Kinesis Firehose metric streams to Sumo Logic.	CloudWatch Metrics, AWS Observability	Docs
AWS Kinesis Firehose for Logs	This project sets up AWS resources to Send AWS CloudWatch Logs using Kinesis Firehose streams to Sumo Logic.	CloudWatch Logs, AWS Observability	Docs
Kinesis to Sumo Logic	This function is invoked by AWS Lambda after it detects new records in Kinesis stream. The received collection of events are decompressed, transformed and send to Sumo Logic HTTP source endpoint		Docs
SumoLogic Lambda Function for AWS CloudWatch Logs	This function subscribes to CloudWatch Log Group and is invoked by AWS CloudWatch with log messages as payload. The records received are decompressed, transformed and forwarded to Sumo Logic HTTP source endpoint in chunks.While the function is more simple then the DLQ-based solution above, it doesn't handle failures and retries properly, thus not recommended.	Not Recommended	Docs
S3 AND Cloudtrail S3 to Sumo Logic	This function receives S3 notifications on new files uploaded to the source S3 bucket, then reads these files, unzips them, and breakdown the records before finally sending to HTTP hosted collector endpoint.	DEPRECATED	Docs Docs

Helper Functions

FunctionName	Description	Setup Documentation
Kinesis Firehose Processor	This function is used for transforming streaming data from kinesis firehose before it sents to destination.	Docs
LogGroup Lambda Connector	This function is used to automatically subscribe newly created and existing Cloudwatch LogGroups to a Lambda function.	Docs

Supported Runtimes

• All the nodejs functions are tested with nodejs runtime 4.3 and 8.10.

• All the python functions are tested with python version 2.7.

Testing with TravisCI

• All the test are currently in python and travis.yml is configured to run any file with prefix "test_" present in lambda function's folder.

• All the dependencies(defined in package.json) of lambda function are installed first and then build is created.

• For adding test for new function you need to specify FUNCTION_DIR(lambda function's folder) and node_js(node js version) under jobs field in travis.yml. This is done because currently testing same function in parallel with different node versions throws function resource exists error (name collision) and therefore are run sequentially.

TLS 1.2 Requirement

Sumo Logic only accepts connections from clients using TLS version 1.2 or greater. To utilize the content of this repo, ensure that it's running in an execution environment that is configured to use TLS 1.2 or greater.

sumologic-aws-lambda's Issues

Slow Log Processing with DLQ Lambda

I'm seeing very slow processing of the CloudWatch dead letter queue by the SumoCWProcessDLQLambda.

The logs repeatedly show errors similar to this one:

2019-05-03T16:05:44.119Z bfc53687-23d5-4155-87f8-1804134eabfa TypeError: Cannot read property 'data' of undefined at /var/task/DLQProcessor.js:19:73 at Response.<anonymous> (/var/task/DLQProcessor.js:41:19) at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:364:18) at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:109:20) at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:81:10) at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:683:14) at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12) at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)

It appears there is some data in the queue that is causing the lambda to get hung up. Previously, the queue could be emptied very quickly.

Will the cloudwatchlogs_lambda.js lambda support TLS 2.1

Hi,

Sumologic is moving its http endpoints to TLS 1.2 so I wanted to ask if there would be any change in the cloudwatchlogs_lambda.js code to accomodate that? Sorry I am not a node/js developer but I am mainting a project that uses your code. I have an idea of what change might be required to explicitly provide the version but I cannot find what the default bahaviour is.

Sumologic will stop supporting older version on 20th of June of this year.

Thanks!

cloudwatchlog_lambda.js creates a log loop by sending logs that indicate a log has been sent to SumoLogic

When a log is sent to SumoLogic by this Lambda, a log is created. This log is then sent by the Lambda and a new log is created. This behavior is a log loop and should be guarded against by the Lambda's logic.

Example message:

message:"[2024/04/12 14:36:21] [ info] [output:http:http.1] endpoint5.collection.us2.sumologic.com:443, HTTP status=200",

Any plan for a new release? The last one (v1.2.8) was created in Oct 2019.

Hi,

Code on master already supports node14, however, the latest release (v1.2.8) is still node8 which is NOT supported by AWS Lambda.

To stay a valid open-source project, I believe the release tag needs to catch up.

Thanks!
Tony

Estimates, when the new release can be expected

Hi guys,

Any estimates when the new release can be expected ? We are automating build process to use your lambda cloudwatchlogs function and would like to rely on tag number rather then just copy stuff from master.

Thanks

nodejs12 support?

Hello,

Are there plans to upgrade the nodejs runtime from 10 -> 12?

No data displaying in SumoLogic

Running "cloudwatchlogs_lambda.js" as-is wasn't displaying any logs when searching sumologic. I added "toString" to the request, e.g.:
res.on('data', function(chunk) { body += chunk.toString(); });
on line 32 and now see logs. I'm not super familiar with nodejs, but this seems to have worked for me.

Documentation Link Broken

It would have been tremendously helpful to have this link included in the README, as the original link in the lambda script points to a place that didn't work for me:

http://help.sumologic.com/Apps/AWS_Lambda/Collect_Logs_for_AWS_Lambda?t=1461360129021

Add to awesome-functions

It's a good candidate for https://github.com/plutov/awesome-functions

Null RequestIDs created by function for Lambda logs

The requestID is set to be null for most of the time for each execution segment. It should be extracted from the first line when the execution starts, and persisted through out.

errors is not defined, messageList never cleared

The variable errors in cloudwatchlogs_lambda.js line 121 is not defined.
When the http connection fails it would be nice to know the http response code.

Support arm64 runtime and Node16?

Any plans to validate and support the code to run on ARM64 and Node16 runtime?
ARM64 will help reduce customer operational costs.

https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

How to Set Up Sumo Logic App for Amazon VPC Flow Logs

Hi, for lack of a better place to ask this, I came across this github while googling. How in the world do you set up the "Sumo Logic App for Amazon VPC Flow Logs" application, referenced in Sumologic docs here: https://service.sumologic.com/help/Default.htm#Amazon_VPC_Flow_Logs_App.htm%3FTocPath%3DApps%7CSumo%2520Logic%2520App%2520for%2520Amazon%2520VPC%2520Flow%2520Logs%7C_____0

Is that what this lambda streaming setup described in this github for? Any help or documentation would be much appreciated!

cloudwatchlogs_lambda should send a batch of messages to Sumo instead of sending them one by one

CloudWatch supports passing several log messages to the streaming lambda.

In many cases CloudWatch passes several log messages in one event. Current implementation of cloudwatchlogs_lambda.js splits the messages and sends them to Sumo HTTP collector one by one.

Why not send all the messages in a single HTTP request (similar to what Log4J Sumo adapter does)?

Subscription not spelled correctly in CF template

SumoCWLogSubsriptionFilter should be SumoCWLogSubscriptionFilter

loggroup-lambda-connector: connect existing log groups?

Hello!

I was wondering if there's a good way to subscribe existing log groups in addition to auto-subscribing new log groups that match the regex?

Maybe even just an example of how to invoke the lambda manually to subscribe if that's possible?

using _sumo_metadata with non json logs

hello, we'd like to use _sumo_metadata, but we currently send our logs as text only. would it be possible to send a message like this:

{
  _sumo_metadata: {...},
  message: 'stuff'
}

and have the log appear as just 'stuff' as text, instead of {message: 'stuff'} on sumo? if not, then would you be open to adding a flag and some code to facilitate it?

Thanks!

Tests do not seems to be working.

Running the tests seems to be problematic when following the readme.

First, I needed to specify an environment variable export SumoEndPointURL=https://endpoint2.collection.us2.sumologic.com/receiver/v1/http/Zxxxxxxxxx...

Even with that in place, it appears to be hardcoded to attempt to upload the content to the bucket 'appdevstore-us-east-2'

Add an overflow option for lambda functions when data ingestion is throttled by Sumo.

Right now data is usually saved into the running lambda log group. We should support saving overflow data into S3.

support for unicode characters for logging

The data extracted from the stream to be sent to sumo is being done in 'ascii' vs 'utf-8' so the lambda fails when a log statement contains unicode (see offending code below)

var awslogsData = JSON.parse(buffer.toString('ascii'));

cc @jefftrudeau

Support China region deployment

Dear team:

Does this project have plan to support the deployment within AWS China regions?
Thanks

Node.js 8.10 is EOL, please migrate your functions to a newer runtime version

Any immediate plans to upgrade the runtime node version?

Kinesis Function produces duplicate messages

messageList needs to be empty before each run through the record loop.

https://github.com/SumoLogic/sumologic-aws-lambda/blob/master/kinesis/node.js/k2sl_lambda.js#L139

2017-10-16T23:32:45.844Z	d3a969ca-3cdd-4639-bcc6-7f603a0affcd	Kinesis Records: 3
2017-10-16T23:32:45.890Z	d3a969ca-3cdd-4639-bcc6-7f603a0affcd	Log events: 997
2017-10-16T23:32:45.928Z	d3a969ca-3cdd-4639-bcc6-7f603a0affcd	Log events: 997
2017-10-16T23:32:45.968Z	d3a969ca-3cdd-4639-bcc6-7f603a0affcd	Log events: 997
2017-10-16T23:32:46.447Z	d3a969ca-3cdd-4639-bcc6-7f603a0affcd	messagesSent: 1 messagesErrors: 0
2017-10-16T23:32:46.803Z	d3a969ca-3cdd-4639-bcc6-7f603a0affcd	messagesSent: 1 messagesErrors: 0

To test this I've created a paragraph-sized message which gets spit out 10000 times. If the logs are smaller you don't see this error. In my test, referencing the log above, I get 2x messages after message 997 in sumo and 3x messages after 1994.

Fix is simple. Declare messageList inside the loop

Feature Request: Retries

@g-wattz We use cloudwatchlogs_lambda.js to send CloudWatch Logs to SumoLogic. We found that this Lambda fails intermittently and the response was {"errorMessage":"errors: HTTP Return code 504"}. This looks like SumoLogic is having trouble consuming the log message in these occasions.

Is there a plan to implement retries based on this error code?

Thanks.

AWS Lambda: Node.js v4.x is now EOL

AWS is removing support for Node.js v4.x.

New Lambda functions using Node.js v4.x will no longer be able to be created on July 31st, 2018.
Existing Lambda functions will no longer allow code updates on October 31st, 2018.

Requesting project to be updated to a newer Node.js version, preferably v8.

Security Hub Collector and Forwarder are both missing from the Serverless Application Repository

I'm attempting to setup the collector and forwarder for AWS Security Hub. The documentation(https://help.sumologic.com/07Sumo-Logic-Apps/01Amazon_and_AWS/AWS_Security_Hub/1-Ingest-findings-into-AWS-Security_Hub and https://help.sumologic.com/07Sumo-Logic-Apps/01Amazon_and_AWS/AWS_Security_Hub/2-Collect-findings-for-the-AWS-Security-Hub-App) sends you to the Serverless Application Repo for both of those apps, but neither of them show up when I search for sumo. I checked in multiple regions to make sure it wasn't a region thing, but got the same results.

CloudFormation stack fails

The CF stack launched by this application fails to complete, rolling back with an error that the IAM role aws-serverless-repository-CloudWatchEventFunctionR-1IATRDMOBCJVM doesn't exist

Update deprecated Lambda runtime for SecurityHubCollectorFunction

The Python 3.7 Lambda runtime has been deprecated, but the SecurityHubCollectorFunction is still using it.

sumologic-aws-lambda/securityhub-collector/sam/template.yaml

Line 20 in 305e7cf

Runtime: python3.7

Potential SAR Vulnerability

Hi @SumoSourabh

The Readme.md for the CloudWatchEvents deployments needs an update, as it contains an AWS SAR security vulnerability that has been recently discovered. We wrote a detailed explanation on the vulnerability.

It is important to add a link to the source account that deploys to the bucket by adding an additional condition.

            Condition:
              StringEquals:
                "aws:SourceAccount":  <AWS::AccountId>

@SumoSourabh I am tagging you as I have seen that you recently modified the file.

Documentation

https://docs.google.com/document/d/1vxdXSPetE2_F9e8K17KygSWQVZbTzveX/edit?usp=share_link&ouid=114724101433132334742&rtpof=true&sd=true

Cannot process Nodejs10 Cloudwatch logs

The log level that is now included with logs for the Node10 lambdas, means that json logs can not be parsed correctly.

kinesis firehose processor

I have installed the kinesis firehose processor but seeing weird characters in SumoLogic.

I have used the readme from the following github url https://github.com/SumoLogic/sumologic-aws-lambda/tree/master/kinesisfirehose-processor to implement the kinesis firehose processor.

When collecting the data from S3 in SumoLogic I see the following result. Please help figuring out what is going on...

�5�[��0��
��+v|ޢ�ݶ���D�@q5e"��fQ����ۇy9gt��w#�O��7x� _����sY��˒�I����d���a܈<����2��!9^Cv��1�MW�H�Xӡ#����m��.c�w�&����)ߘߠ�l����GrG�m�������l��*�eY��9�P�S��BsTRjt���"�Ņ��S�ss�d���O���C"�G���wq�o�9�J!5�\HõB��R0�Ų4`�)%� �R(51�M"�MPxRr�2���?�_Vź����KZ�\̸���Բ<��:O������0ۤ�/��'�������q$��

First event into new logGroup won't invoke the subscribing function.

If new log group creation is binded with new event creation, for example, in case of first run of new lambda function. Only after creation of Log group, event rule will be invoked to attach it to target lambda function.
So first event will not be captured in target lambda function. Only subsequent events will be captured.

Support filter pattern in loggroup lambda connector

Currently the filter pattern is default to empty ('') and not configurable.

async function createSubscriptionFilter(lambdaLogGroupName, destinationArn, roleArn) {
    if (destinationArn.startsWith("arn:aws:lambda")){
        var params = {
            destinationArn: destinationArn,
            filterName: 'SumoLGLBDFilter',
            filterPattern: '',
            logGroupName: lambdaLogGroupName
        };
    } else {
        var params = {
            destinationArn: destinationArn,
            filterName: 'SumoLGLBDFilter',
            filterPattern: '',
            logGroupName: lambdaLogGroupName,
            roleArn: roleArn
        };
    }

Can we add support to define our filter pattern so that we can selectively ingest the logs?

Thanks

The containing application will need be updated before it can be deployed

Hi,

I'm getting the following error when trying to deploy the AWS Lambda function for "sumologic-guardduty-benchmark — version 1.0.5"

The following nested application arn:aws:serverlessrepo:us-east-1:956882708938:applications/sumologic-app-utils and semantic version 1.0.5 was recreated after the containing application arn:aws:serverlessrepo:us-east-1:956882708938:applications/sumologic-guardduty-benchmark was published. The containing application will need be updated before it can be deployed

Please advise/fix.

Regards,
Suhail.

Dead link in cloudwatchlogs README.md

The deprecation warning in cloudwarchlogs/README.md links to a page which does not exist

https://github.com/SumoLogic/sumologic-aws-lambda/blob/main/cloudwatchlogs/README.md?plain=1#L5-L6

Steps to reproduce:

https://help.sumologic.com/Send-Data/Collect-from-Other-Data-Sources/Amazon-CloudWatch-Logs

Proposed resolution:

Update link with correct documentation.

cloudwatch_lambda.js uses old, deprecated `context` object

The Problem

In the current cloudwatch_lambda.js (pinned version link), the lambda is being marked as complete by calling context.succeed:

sumologic-aws-lambda/cloudwatchlogs/cloudwatchlogs_lambda.js

Line 96 in 22a1dcb

context.succeed();

However, this is very old syntax. It was deprecated with the 0.11.x runtime and backfilled (poorly) into the new runtimes.

Since the docs now recommend running Node 8.10, this function should be rewritten using the more understandable async/await syntax. Here are the docs stating that it's available in the 8.10 runtime.

Rewriting the function to use async await will transition the runtime to use the explicit callback parameter, marked by when the function resolves the promise returned by the exports.handler method.

How I Discovered This

Because we're defining this lambda in our Infrastructure-as-Code repository (using Terraform), we encrypt the SUMO_ENDPOINT URL since we don't want anyone other than the lambda to be able to see it (a malicious actor could POST logs to the endpoint and run up our bill).

To do this, I made the exports.handler asynchronous and decrypt the KMS key before parsing the message and sending it to sumo.

const AWS = require('aws-sdk');
async function getSumoUrl() {
  var kms = new AWS.KMS();
  var kmsDecryptParams = {
    CiphertextBlob: Buffer.from(process.env.ENCRYPTED_SUMO_ENDPOINT, 'base64')
  };
  var data = await kms.decrypt(kmsDecryptParams).promise();
  var decryptedSumoEndpoint = data.Plaintext.toString('utf-8');
 
  return decryptedSumoEndpoint;
}

// existing code unchanged

// NOTE that this is async now
exports.handler = async function (event, context) {
  SumoURL = await getSumoUrl();

  // existing code unchanged
}

However, this did not work because of the reliance on the old, unsupported context.succeed call. This would have been a whole lot easier if the rest of the function used the newer Node 8 async/await syntax to start with.

Conclusion / Suggestions

It seems like this lambda was written a long time ago, based on the APIs it's using. Node has come quite a long way and rewriting with Node 8 standards would make it more readable and extensible for users.

Linting CloudFormation templates would make them simpler

Have you considered using AWS CloudFormation linter https://github.com/aws-cloudformation/cfn-lint against the CloudFormation templates in this repository:

• https://github.com/SumoLogic/sumologic-aws-lambda/blob/main/cloudwatchlogs-with-dlq/DLQLambdaCloudFormation.json
• https://github.com/SumoLogic/sumologic-aws-lambda/blob/main/cloudwatchlogs-with-dlq/DLQLambdaCloudFormationWithSecuredEndpoint.json

in order to improve them?

Regex is not working

This doesn't seem to be working:

// Regex used to detect logs coming from lambda functions.
// The regex will parse out the requestID and strip the timestamp
// Example: 2016-11-10T23:11:54.523Z	108af3bb-a79b-11e6-8bd7-91c363cc05d9    some message
var consoleFormatRegex = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z\t(\w+?-\w+?-\w+?-\w+?-\w+)\t/;

A simple message like this: "START RequestId: be746bac-efd6-11e7-9740-0f2037ef62c0 Version: $LATEST\n"; is returning null.

Update descripttion of logstream prefix

"LogStreamPrefix": {
"Type": "String",
"Description": "(Optional) Enter comma separated list of logStream name prefixes to filter by logStream. Please note this is seperate from a logGroup. This is used to only send certain logStreams within a cloudwatch logGroup(s). LogGroups still need to be subscribed to the created Lambda funciton, regardless of what is input for this value.",
"Default": ""
}

not defined warning

I'm getting final_event is not defined warning in lambda for cloudwatchevents.js code

Any way to honor the JSON Object in val.Message and not stringify it?

Confusing title, but our Lambda functions are logging valid JSON objects and not strings like so:

{
    "fields": {
        "MediaPath": {
            "Path": "1466672936684.jpg",
        },
        "RequestID": "a7188dc9-6968-11a6-be87-3326b0d7d49d",
        "app": "purge"
    },
    "level": "info",
    "timestamp": "2016-08-23T19:36:39.382610474Z",
    "message": "success"
}

When it's passed over to cloudwatchlogs_lambda.js Sumo then sees this:

{
   "id":"32826273204296444651588478853452971566044640956554084352",
   "timestamp":1471980999382,
   "message":"{\"fields\":{\"MediaPath\":{\"Path\":\"1466672936684.jpg\"},\"RequestID\":\"a7188dc9-6968-11a6-be87-3326b0d7d49d\",\"app\":\"purge\"},\"level\":\"info\",\"timestamp\":\"2016-08-23T19:36:39.382610474Z\",\"message\":\"success\"}
",
  "requestID":null,
  "logStream":"2016/08/23/[$LATEST]8e1c1bde6beb44df800fb15f6444263a",
  "logGroup":"/aws/lambda/purge"
}

Which doesn't make it ideal to set up search queries in Sumo. Is there anyway to persist the JSON object through and have the "message" be JSON

Lambda connector - create subscription fails due to throttling

When there are a large number of existing log groups, and USE_EXISTING_LOG_GROUPS=True then the putSubscriptionFilter AWS API call sometimes fails due to hitting throttling limits. We see errors similar to the one below:

2020-01-21T12:23:02.436Z	a30ab1f3-8cfe-4f5c-84e4-0d404da581cc	INFO	Error in subscribing /aws/lambda/dev-somefunction { LimitExceededException: Resource limit exceeded.
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:51:27)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
  message: 'Resource limit exceeded.',
  code: 'LimitExceededException',
  time: 2020-01-21T12:23:02.377Z,
  requestId: 'a2066014-38a4-43b3-934e-0309185b910f',
  statusCode: 400,
  retryable: false,
  retryDelay: 38.911558105243586 }

Re-trying the invocation usually causes the throttle limits to be hit in the same places, so doesn't help to resolve the situation.