sbstjn / sqs-worker-serverless

Example for SQS Worker in AWS Lambda using Serverless

SQS Worker with Serverless ⚡️

Experimental Serverless Amazon SQS Worker with AWS Lambda.

Update: Please see Serverless Analytics for a more realistic setup using Amazon Kinesis Streams to process your data and events. You can read more about the Kinesis and Lambda connection at Serverless Analytics with Amazon Kinesis and AWS Lambda. This was only intended as an April Fools' project …

Setup

• SQS Queue with your messages
• SNS Topic to handle CloudWatch Alarms
• DynamoDB table to persist configuration
• CloudWatch Schedule as cron replacement
• Three (scale, worker, process) AWS Lambda functions

Workflow

• CloudWatch Alarms on queue length post to SNS
• SNS Topic triggers scale Lambda function
• Function scale updates configuration in DynamoDB
• CloudWatch Schedule invokes worker every x minute(s)
• Function worker reads configuration from DynamoDB
• Function worker invokes process function(s)

Auto-Scaling with CloudWatch Alerts

Workers with CloudWatch Schedule

Deploy

$ > yarn install
$ > yarn deploy

Add noise to SQS

You should have some data in your queue to test this setup. Use wrk to send messages to SQS, but make sure to enable anonymous access to sendMessage for your queue first!

$ > wrk -c35 -d60 -t35 \
    -s helpers/wrk.lua \
    https://sqs.REGION.amazonaws.com/ACCOUNT-ID/YourQueueName

sqs-worker-serverless's Issues

examples repo

@sbstjn you should add this project here! https://github.com/serverless/examples

Reduce worker costs when queue is empty

Hello,

actually the worker-function is also invoked when the queue is empty. I think it would be better to control the active field of the rate-rule depending on a cloudwatch alarm too.

Inital: Rate is deactivated
Zero Messages => Deactive AWS::Events:Rule
One or more Messages in the Queue => Activate AWS::Events:Rule

What do you think?

Scale up Cloudwatch Alarm Regex typo

Hey man, thanks for putting this together! I love how it works.

Just noticed when I tried to port this to my project that the scale up Cloudwatch alarm regex has a typo: https://github.com/sbstjn/sqs-worker-serverless/blob/master/functions/scale.js#L7

The Cloudwatch alarms that were set up for me had Message being singular instead of plural.

Maybe I misconfigured something but I believe we should update:

 const regex = /(OK|ALARM):(.*)MessagesAlarm([0-9]*)/g

to

 const regex = /(OK|ALARM):(.*)MessageAlarm([0-9]*)/g