pagerduty / pd-oncall-chat-topic

AWS Lambda Function that updates a Chat Room topic (eg, Slack)

pd-oncall-chat-topic

AWS Lambda Function that updates a Chat Room topic (eg, Slack)

Motivation

At PagerDuty, we use PagerDuty to manage our on-call schedules. One integration that we like is knowing who is on-call for a given team, via posting the information in a Slack channel topic.

At PagerDuty Summit, one of our customers asked if we have this integration open-sourced for them to use as well. At the time, we did not, it was deeply integrated into our ChatOps tooling that is very specific to the PagerDuty infrastructure. However, this integration is applicable to many other organizations.

This project started at one of our Hack Days, a day put aside monthly for employees to build and present projects or ideas that fulfill some kind of need at the company.

How to Deploy

1. Create an Integrated Bot that you can invite to your channel. https://my.slack.com/services/new/bot
2. Obtain a PagerDuty API Key (v2) Directions Here
3. Deploy CloudFormation

• Clone repo
• Modify 2 variables for your AWS Environment (Makefile#L1-L2)
• make deploy

4. Write API Keys to EC2 SSM Parameter Store.

• The lambda function expects certain key names by default so the following commands should work unless modified elsewhere (advanced config).
• make put-pd-key
• make put-slack-key

5. Write Config to DynamoDB for which channels to update.

• It is possible to use the AWS CLI for this (or finish #4 for ease of use)
• In lieu of above, manually update the table with item entries of this format:

{
  "schedule": "P123456",
  "slack": "C123456",
  "sched_name": "Optional schedule name to use in topic"
}

(where schedule is the PagerDuty Schedule ID, and slack is the Slack Channel ID. You can have a space-separated list of channels. sched_name is optional and if omitted will be looked up) If you have a split on-call rotation, you may place multiple comma-separated schedules and schedule names.

Architecture

The main part of this infrastructure is an AWS Lambda Function that operates on a schedule (cron), reads configuration information from an DynamoDB Table and secrets from AWS EC2 Parameter Store. This is all deployed from a AWS CloudFormation template.

Cost

The way that this Lambda Function is configured, is to run on a schedule every 5 minutes. Some basic (anecdotal) testing revealed that the execution is about 5 seconds per 5 updates (via threading). Assuming double that and erroring on the side of a large configuration (10x) the execution time will cost below $2/month. The DDB table will cost, ~$0.60/month.

Contact

This integration is primarily maintained by the SRE Team at PagerDuty. The best way to reach us is by opening a GitHub issue.

pd-oncall-chat-topic's Issues

Bug: Does not support #channel in the topic

The bot doesn't parse #channel in the topic properly. See example workarounds here:

pd-oncall-chat-topic/lambda/main.py

Lines 113 to 120 in ca3e55c

	# Because Slack adds a '<mailto:[email protected]|[email protected]>' behind the
	# scenes, we need to match the email address in the first capturing group,
	# then replace the rest of the string with the address
	current_full_topic = re.sub(r'<mailto:([a-zA-Z@.]*)(?:[|a-zA-Z@.]*)>',
	r'\1', get_slack_topic(channel))
	# Also handle Slack "Subteams" in the same way as above
	current_full_topic = re.sub(r'<(?:!subteam\^[A-Z0-9|]*)([@A-Za-z-]*)>', r'\1',
	current_full_topic)

Support Stride

HipChat's replacement

issue raised in wrong repo, ignore

Add to serverless repository

It'd be nice if this was available to install directly from the AWS Serverless Repository

Support HipChat (cloud)

There is some initial stub to support HipChat.

This task is to finish that out. https://www.hipchat.com/docs/apiv2/method/set_topic

Admin: Add cli helpers to make it easy to add config info

Write some script that will take user input and add it to the proper DDB table. You can use cloudformation output to find the DDB table, example here: 0d29d94

Automatically clean up DDB Table after too many failures

Things in PD or Slack may change. In an effort to save execution time, it might make sense to log how many failures there are and then after N failures, remove the ddb entry.

The DDB table already supports TTL so the logic could be extended to log the failures then add a TTL entry.

how to test after following the deploy steps?

Hi,

I've configured all the steps up to step number 5, but i'm not sure if it's working, i tried change the on-call schedules from PagerDuty but there are no alerts that pop-up.

Is there a way to test this from AWS Lambda or DynamoDB?

Thanks, hope to hear from you soon...