miznokruge / sqs-notify

AWS SQS listener

SQS notify

Listen a SQS queue, execute a command when received. A message body is passed as STDIN to the command.

(in Japanese) SQS を監視しメッセージを取得したら指定のコマンドを実行します。 メッセージの内容はコマンドの標準入力として渡します。

Installation

$ go get github.com/koron/sqs-notify

or update

$ go get -u github.com/koron/sqs-notify

Require git command for dependencies.

for developer

$ go get github.com/goamz/goamz

Usage

From online help.

Usage: sqs-notify [OPTIONS] {queue name} {command and args...}

OPTIONS:
  -region {region} :    name of region (default: us-east-1)
  -worker {num} :       num of workers (default: 4)
  -nowait :             didn't wait end of command to delete message
  -ignorefailure:       don't care command results, treat it as success always
  -retrymax {num} :     num of retry count (default: 4)
  -msgcache {num} :     num of last received messages in cache (default: 0)
  -redis {path} :       use redis as message cache (default: disabled)
  -logfile {path} :     log file path ("-" for stdout)
  -pidfile {path} :     pid file path (available with -logfile)

  -mode {mode} :        pre-defined set of options for specific usecases

Source: https://github.com/koron/sqs-notify

Guide

Basic usage:

sqs-notify [-region {region}] {queue name} {command and args}

1. Prepare AWS auth information.

   1. Use ~/.aws/credentials (recomended).

      sqs-notify supports ~/.aws/credentials file, and use info from sqs-notify or default section in this order. Example:

      [sqs-notify]
      aws_access_key_id=foo
      aws_secret_access_key=bar

   2. Use two environment variables.

      • AWS_ACCESS_KEY_ID
      • AWS_SECRET_ACCESS_KEY

2. Run sqs-notify

   $ sqs-notify your-queue cat
   

   This example just copy messages to STDOUT. If you want to access the queue via ap-northeast-1 region, use below command.

   $ sqs-notify -region ap-northeast-1 your-queue cat
   

Name of regions

• us-east-1 (default)
• us-west-1
• us-west-2
• eu-west-1
• ap-southeast-1
• ap-southeast-2
• ap-northeast-1
• sp-east-1

Logging

When -logfile {FILE PATH} is given, all messages which received are logged into the file. If FILE PATH is -, it output all logs to STDOUT not file.

Using -pidfile {FILE PATH} with -logfile, sqs-notify writes own PID to the file. You can send SIGHUP to that PID to rotate log.

Log example:

2014/05/28 00:13:29 EXECUTED queue:kaoriya body:"abc\n" cmd:cat status:0
2014/05/28 00:14:27 EXECUTED queue:kaoriya body:"def\n" cmd:cat status:0
2014/05/28 00:14:54 EXECUTED queue:kaoriya body:"foo\nbar\n" cmd:cat status:0

Each items in one log are separated by tab char (\t).

Miscellaneous

LF at EOF

When message doesn't have LF at EOF (end of file/message), the last line can't be handled by read shell command or so. This is limitation of read command, not sqs-notify. Therefore this kind of scripts don't work correctly for messages without LF at EOF:

#!/bin/sh
while read line
do
  echo "received: $line"
done

To work around this problem, use xargs like this.

#!/bin/sh
xargs -0 echo | (
while read line
do
  echo "received: $line"
done
)

/dev/stdin

You can use /dev/stdin pseudo file, if your system support it, like this:

#!/bin/sh

data=`cat /dev/stdin`

# do something for data.

sqs-echo

sqs-echo is useful for debugging received SQS message with sqs-notify. It just shows date, time, byte num and contents of received messages. Example output is below:

2015/05/07 12:43:08 (7) "foo\nbar"
2015/05/07 12:43:12 (3) "qux"

You can install sqs-echo with below command.

$ go install github.com/koron/sqs-notify/cmd/sqs-echo

Suppress for duplicated message

To suppress executions of the command for duplicated (seen recently) messages, please use -msgcache {num} option. {num} means cache size for messages.

Below command shows how to enable suppression with cache size 10.

$ sqs-notify -msgcache 10 -logfile - my-queue sqs-echo

You'll get log like this when sqs-notify detect duplicated message.

2015/05/07 12:13:08 EXECUTED queue:my-queue body:"foo\n" cmd:cat status:0
2015/05/07 12:43:11 SKIPPED  queue:my-queue body:"foo\n"

sqs-notify stores hashs (MD5) of message to detect duplication.

Use redis to suppress duplicated messages

sqs-notify では -redis オプションに設定ファイル(JSON)を指定すると、メッセージ ハッシュの保存先をRedisに変更できます。

実行例:

$ sqs-notify -redis redis-example.json -logfile - my-queue sqs-echo

JSON内で使用できる設定項目は以下のとおりです。

Name	Mandatory?	Description
addr	YES	"host:port" address
keyPrefix	YES	prefix of keys
expiration	YES	time to expiration for keys (acceptable format)
network	NO	"tcp" or "udp" (default "tcp")
password	NO	password to connect redis
maxRetries	NO	default: no retry

redis-example.json には最小限の設定が記載されています。

at-most-once mode

at-most-once mode provides at-most-once command execution.

$ sqs-notify -mode at-most-once -msgcache 10 my-queue sqs-echo

This mode implies -ignorefailure option, excludes -nowait option, and requires one of -msgcache or -redis option.