guardian / podcasts-analytics-lambda

An AWS Lambda to monitor podcasts consumption 🎵

podcasts-analytics-lambda

This lambda is responsible for sending podcasts metrics to Ophan, our real time analytics system.

It does so by mining our CDN logs (stored in an S3 bucket), collecting information about every single file hit and constructing a model out of every log line. If a log line is parsed correctly into the model, an event is sent to Ophan mimicking a pageview. Ophan has some ad-hoc logic and knows how to handle these special requests.

This lambda currently pulls audio data from logs provided by Fastly and Acast.

Architecture overview

+---------------+      +------------+                 +--------+                       +-------+
| Fastly/Acast  | ---> | S3 bucket* | -- triggers --> | Lambda | -- send events to --> | Ophan |
+---------------+      +------------+                 +--------+                       +-------+

• The fastly and acast logs arrive in separate buckets, and another lambda copies them into a single bucket. Any log files above a certain size will be split into several files to ensure this lambda does not run out of time.

Run & Tests

To run locally you need to export a valid PROD CAPI key as a local variable:

export CAPI_KEY='your-PROD-key'
sbt test

Monitoring

The lambda logs warnings and errors to cloudwatch. E.g. a failed capi request is ERROR level, and failure to parse a single line is WARN level.

Cloudwatch alarms are triggered if:

• the lambda invocation rate drops below 200/hour
• the lambda fails with an error (e.g. out of memory, timeout)
• the lambda logs any errors or warnings.

podcasts-analytics-lambda's Issues

Update other build configuration

The master branch of this repository has been migrated to main using the master-to-main tool.

Please check any build related configuration and update as required:

• TeamCity - See the required steps in the migrating.md document
• Change snyk github integration(s) - it uses the default branch, but you will need to delete and reimport the project+file as this is the only way to refresh the default branch at present.
• Any other externally configured analysis tooling your team is using e.g. travis CI

It's probably a good idea to merge test PR to main once this is complete, to make sure that everything is working as expected. 🙂

Update Riff Raff configuration

The master branch of this repository has been migrated to main using the master-to-main tool.

The following riff-raff.yaml file(s) have been found in the repostiory:

• riff-raff.yaml

For each deployment, you will need to complete the following steps:

1. Fix continuous deployments: https://riffraff.gutools.co.uk/deployment/continuous
2. Ensure blocked deployments are still blocked: https://riffraff.gutools.co.uk/deployment/restrictions