awslabs / aws-streaming-media-analytics

AWS Streaming Media Analytics is a serverless end-to-end solution for analyzing the video streaming experience

AWS Streaming Media Analytics

AWS Streaming Media Analytics is a serverless end-to-end solution for analyzing the video streaming experience

• AWS Streaming Media Analytics
  • Launching from a Pre-Built template
  • Reinvent Workshop
  • Solution Architecture
  • How to customize and create your own CloudFormation template.
  • JavaScript Video Player Setup
  • License Summary

Launching solution with Pre-built AWS CloudFormation Template

The solution is deployed using an AWS CloudFormation template with AWS Lambda backed custom resources. To deploy the solution, use one of the following CloudFormation templates and follow the instructions below.

AWS Region	AWS CloudFormation Template URL
ap-northeast-1 (Tokyo)	Launch stack
ap-northeast-2 (Seoul)	Launch stack
ap-south-1 (Mumbai)	Launch stack
ap-southeast-1 (Singapore)	Launch stack
ap-southeast-2 (Sydney)	Launch stack
eu-west-1 (Dublin)	Launch stack
eu-west-2 (London)	Launch stack
us-east-1 (N. Virginia)	Launch stack
us-east-2 (Ohio)	Launch stack
us-west-2 (Oregon)	Launch stack

This CloudFormation template deploys an example OTT streaming platform in to your AWS account. The deployed environment includes a static website hosted on S3 with sample videos, Kinesis Data Firehose endpoints to accept and process streaming log data and a real-time log analysis application built on Kinesis Analytics, DynamoDB and AppSync.

If you want to make changes to the code change the S3Bucket field in the mapping section of the CloudFormation document to your own bucket. Make sure to put the lambda code in the region you want to deploy.

1. Click on one of the links above to deploy the solution via CloudFormation in your AWS account.

2. The CloudFormation Management Console will load with the template URL pre-filled as a Amazon S3 URL. Click the orange Next button located in the bottom right corner of the console to configure the deployment.

3. By default we have set a stack name of mediaqos. There is no need to change this and we refer to this name throughout the guide when identifying resources. If you do decide to change the stack name, please ensure you only use lower-case letters and digits, and keep the name under 12 characters. The stack name is used to name resources throughout the workshop. Keep the name handy as you will need it from time to time to locate resources deployed by the stack.

4. Once you have decided on a stack name, click Next to continue.

5. On the next step, Configure stack options, leave all values as they are and click Next to continue.

6. On the Review step

a. Check the three boxes under Capabilities and transforms to acknowledge the template will create IAM resources and leverage transforms.

b. Click the Create stack button located at the bottom of the page to deploy the template.

The stack should take around 5-10 minutes to deploy.

Quicksight Update Frequency

By default the Glue Trigger is set for every 4 hours to reduce cost. This can frequency can be increased, but check the AWS Glue console pricing for more details. In my testing AWS Glue cost 0.44 cents per run in this solution.

The GlueJob that writes player data to S3 for Athena and Quick sight is located within the WatchTimeGlueTrigger. Look for the field that looks like this. Schedule: cron(0 */4 * * ? *)

Reinvent Workshop

This solution was originally developed for a workshop at AWS re:Invent 2018. An updated version of the lab guides from this workshop are available at the following link: AWS Streaming Video Analytics Workshop

Architecture

How to customize and create your own CloudFormation template.

Setup Instructions To build with Docker && make

Pre-requisites:

• Install docker for your environment as we will use a Docker container to build
• Install AWS CLI
• Install yarn

The build and deployment process for this project has been tested on both Mac and Linux using the vscode and Cloud9 IDEs respectively.

Build Configuration:

Copy the Makefile.sample file to a file named Makefile. In the new file:

• set bucket variable to reflect the S3 bucket name prefix which will be created within a deployment region. Note the region name will be appended to this prefix.
• optionally, set the s3prefix variable to a prefix you wish to prepend to the path of all artifacts put into the S3 bucket(s).
• set regions variable to reflect one or more AWS regions you want the code artifacts to be copied for CloudFormation deployment.
• set stack_name for the Stack Name to use in the deployment.
• set profile to the AWS CLI profile which has necessary permissions to deploy and create all the resources required.

Commands to manage creation/deletion of S3 buckets:

• To create buckets across regions: make creates3
• To delete buckets across regions: make deletes3

Commands to build and deploy the entire project

• make all
• make deploy

Once the deployment is done you should see the player URL in the Outputs section of the CloudFormation template.

Pricing

Reference the AWS Pricing pages for each service used.

JavaScript Video Player Setup

Guide to Setup JavaScript Video Player

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.

aws-streaming-media-analytics's Issues

Update python functions to 3.x

Support for python2.7 is ending in Lambda. We need to update the python lambda functions to a 3.x runtime so the template remains deployable.

Synchronize documentation with published lab guides

The .md guides in this repo are out of sync with the published lab guides used at public events. The published guides have not been generated from markdown, so resyncing this will require some effort.

Published web guides:

1. self-deployed: https://aws-streaming-media-analytics-workshop.s3.amazonaws.com/index.html
2. deployed via AWS Event Engine: https://aws-streaming-media-analytics-workshop.s3.amazonaws.com/ee/index.html

Creating this issue with a goal of two outcomes:

1. included .md lab guides should 100% match the published web guides
2. a method for generating the published web guides should be included so we don't get out of sync again.

unable to deploy cloudformation template (deployment.yaml)

Hi, i have issues in deploying the cloudformation template. I follow your guide to deploy deployment.yaml but found errors.
Seems like it failed to create Schema (line 970-975) and the error message was the S3 Bucket not found.
Can you help to advise how can i solve this issue?

Runtime ImportModuleErrror for DeployFunction Lambda

{
"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module 'aws-sdk'\nRequire stack:\n- /var/task/index.js\n- /var/runtime/index.mjs",
"stack": [
"Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'",
"Require stack:",
"- /var/task/index.js",
"- /var/runtime/index.mjs",
" at _loadUserApp (file:///var/runtime/index.mjs:997:17)",
" at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1029:21)",
" at async start (file:///var/runtime/index.mjs:1192:23)",
" at async file:///var/runtime/index.mjs:1198:1"
]
}

video playback failing due to CORS permissions in Firefox 71

Playing back video in Firefox 71 on Mac results in errors related to permissions (CORS?):

NotAllowedError: The play method is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://dl0q54uth5jdv.cloudfront.netsample-videos/hls/BigBuckBunny.m3u8. (Reason: CORS request did not succeed).

Fastly Log Processor Function optional in CFN template

We intend to support multiple CDN log formats with this project. Typical use-case should be that each CDN is enabled by a parameter in the CFN template.

CloudFront will be on by default, as we can control that set up in the template, but if we ever break cloudfront into an optional component, we can revisit this.

First step will be to create a toggle for the fast.ly support which will selectively deploy the lambda function and the updated bucket policy for S3. This will be False by default.

nodejs8.10 no longer supported in lambda

lambda has discontinued support for deploying functions with the nodejs8.10 runtime.

Issue to track migrating to a newer runtime.

Update nodejs functions to 12.x

nodejs 8.x and 10.x are ending support. We need to update the lambda functions to 12.x to keep the template functional.

S3 Bucket ACLs being disabled by default (new feature) cause deployment failure

Need to update the S3 buckets in the CloudFormation template to keep using ACLs in order to support CloudFront log delivery:

OwnershipControls:
Rules:
- ObjectOwnership: BucketOwnerPreferred

ref: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html#AccessLogsBucketAndFileOwnership

Update isomorphic-fetch / node-fetch

node-fetch is a dependency of isomorphic-fetch which is used by activeuser-appsync-function. A minor security vuln has been reported in node-fetch and this package needs to be updated.

The query is missing in the doc

https://github.com/awslabs/aws-streaming-media-analytics/blame/master/workshop/LAB3.md#L115

"To query the total views of the video named oceans, enter the following query into the Queries console and click the Play button to execute the query:"

The following query does not exist

Customers would want a new UX for this workshop

Customers told me is difficult to follow this workshop with a laptop, because there is a lot of text and they need to scroll a lot to find the information.

I have showed them this theme used by eksworkshop.com.
https://learn.netlify.com/en/
All told me it will be easier to follow this workshop with this theme and "Automatic next/prev buttons to navigate through menu entries"

CloudFormation stack in README needs to be updated as it's using old node.js runtime

https://github.com/awslabs/aws-streaming-media-analytics/blob/master/workshop/WORKSHOP.md references an old CloudFormation template which is using nodejs6 as it's runtime.

Creating an issue to track fixing this.

Add support for additional regions

The Streaming Media Analytics Workshop currently only supports:

• us-east-1
• us-west-2

Since developing the workshop, additional regions have added support for the variety of services used in this project.

Feature request to add support for additional regions:

• ap-southeast-1
• ap-southeast-2
• ap-northeast-1
• ap-northeast-2
• ap-south-1
• ca-central-1
• eu-west-1
• eu-west-2
• eu-west-3
• eu-north-1
• sa-east-1
• us-east-2