whoigit / webhook-mailbox

Webhook Mailbox

This project configures Amazon Web Services, in particular API Gateway, Simple Queue Service, and Lambda, to create an API endpoint which places incoming requests into a queue ("mailbox").

Requests can be popped off of this queue from behind a firewall and delivered to an internal service, without exposing that service to the Internet.

Install

First, configure a virtual environment and install dependencies:

virtualenv --python=python3 .venv
.venv/bin/activate
pip install -r requirements.txt
command -v rehash && rehash

Configure the AWS command line with your credentials and default region. Other configuration methods, such as environment variables, are available.

Security considerations: Do not leave your AWS credentials lying around in ~/.aws or your shell history file on the server. The configure stage generates credentials with limited privileges that can be safely deployed instead.

Configure

The configure subcommand will automatically provision the necessary AWS resources and return credentials and an HTTP endpoint URL:

$ python webhook-mailbox.py configure
Configured queue webhook-mailbox-388cfb with the following credentials:
  AWS_ACCESS_KEY_ID=AKIATM5TRIWFWDM3I4GX
  AWS_SECRET_ACCESS_KEY=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

URL: https://jofx96r5z4.execute-api.us-east-1.amazonaws.com/prod/

Watch

The watch subcommand watches the queue. When a new message, it will issue a request to the given endpoint URL, copying the HTTP method, headers, URL parameters, and request body. The path is discarded.

AWS_ACCESS_KEY_ID= \
AWS_SECRET_ACCESS_KEY= \
python webhook-mailbox.py watch \
  webhook-mailbox-e05a69 \
  https://server.local/endpoint

Integration with Jenkins

The Generic Webhook Trigger plugin for Jenkins can be used to set up an HTTP endpoint to trigger a build.

Enable the trigger on your Jenkins project. It's a good idea to assign a token to it.

On the repository server, the hook URL should look like:

https://xyzzy.execute-api.us-east-1.amazonaws.com/prod/?token=project-token

For additional configuration, such as to set which branch to build, see these examples.

Then, run the mailbox watcher with the corresponding queue name and the Jenkins trigger invocation URL:

AWS_ACCESS_KEY_ID= \
AWS_SECRET_ACCESS_KEY= \
python webhook-mailbox.py watch \
  webhook-mailbox-e05a69 \
  https://jenkins.local/generic-webhook-trigger/invoke

Docker

A Dockerfile is provided for convenience.

Note that currently -it is required, because otherwise the program will hang and be unresponsive to keyboard input. This is a bug.

docker run \
  -it \
  -e AWS_ACCESS_KEY_ID= \
  -e AWS_SECRET_ACCESS_KEY= \
  -e AWS_DEFAULT_REGION=us-east-1 \
  -e WEBHOOK_QUEUE_NAME=webhook-mailbox-e05a69 \
  -e WEBHOOK_FORWARD_URL=https://jenkins.local/generic-webhook-trigger/invoke \
  whoi/webhook-mailbox

Future

It is possible the configuration could be done through CloudFormation or Terraform more robustly.

webhook-mailbox's Issues

Implement watch mode

There should be a subcommand that polls the queue and when a message arrives, forwards it to a given URL.

# create a session given our region, access key ID, and secret access key
session = boto3.Session(...)
sqs = session.resource('sqs')
queue = sqs.get_queue_by_name(QueueName=...)

# long poll for new messages
while True:
    for message in queue.receive_messages():
        ...

Program hangs in docker without --interactive --tty

For some reason, the program polls forever when run in Docker without -it.

Investigate using Terraform, CloudFormation, etc.

This example does all the work of connecting API Gateway and Lambda:

https://learn.hashicorp.com/terraform/aws/lambda-api-gateway