awslabs / aws-streaming-media-analytics

AWS Streaming Media Analytics is a serverless end-to-end solution for analyzing the video streaming experience

AWS Streaming Media Analytics

15 May 2024: AWS Streaming Media Analytics is no longer maintained or supported due to updates in best practices for this workload. This repository will soon be archived

Please see the following other AWS sample projects for an updated set of examples covering this use-case:

• Improving Video Observability with CMCD and CloudFront blog post
• Cloudfront CMCD Realtime Dashboard repository

The following content is preserved for archive purposes

AWS Streaming Media Analytics is a serverless end-to-end solution for analyzing the video streaming experience

• AWS Streaming Media Analytics
  • Solution Architecture
  • How to customize and create your own CloudFormation template.
  • JavaScript Video Player Setup
  • License Summary

Quicksight Update Frequency

By default the Glue Trigger is set for every 4 hours to reduce cost. This can frequency can be increased, but check the AWS Glue console pricing for more details. In my testing AWS Glue cost 0.44 cents per run in this solution.

The GlueJob that writes player data to S3 for Athena and Quick sight is located within the WatchTimeGlueTrigger. Look for the field that looks like this. Schedule: cron(0 */4 * * ? *)

Architecture

How to customize and create your own CloudFormation template.

Setup Instructions To build with Docker && make

Pre-requisites:

• Install docker for your environment as we will use a Docker container to build
• Install AWS CLI
• Install yarn

The build and deployment process for this project has been tested on both Mac and Linux using the vscode and Cloud9 IDEs respectively.

Build Configuration:

Copy the Makefile.sample file to a file named Makefile. In the new file:

• set bucket variable to reflect the S3 bucket name prefix which will be created within a deployment region. Note the region name will be appended to this prefix.
• optionally, set the s3prefix variable to a prefix you wish to prepend to the path of all artifacts put into the S3 bucket(s).
• set regions variable to reflect one or more AWS regions you want the code artifacts to be copied for CloudFormation deployment.
• set stack_name for the Stack Name to use in the deployment.
• set profile to the AWS CLI profile which has necessary permissions to deploy and create all the resources required.

Commands to manage creation/deletion of S3 buckets:

• To create buckets across regions: make creates3
• To delete buckets across regions: make deletes3

Commands to build and deploy the entire project

• make all
• make deploy

Once the deployment is done you should see the player URL in the Outputs section of the CloudFormation template.

Pricing

Reference the AWS Pricing pages for each service used.

JavaScript Video Player Setup

Guide to Setup JavaScript Video Player

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.

aws-streaming-media-analytics's Issues

The query is missing in the doc

https://github.com/awslabs/aws-streaming-media-analytics/blame/master/workshop/LAB3.md#L115

"To query the total views of the video named oceans, enter the following query into the Queries console and click the Play button to execute the query:"

The following query does not exist

CloudFormation stack in README needs to be updated as it's using old node.js runtime

https://github.com/awslabs/aws-streaming-media-analytics/blob/master/workshop/WORKSHOP.md references an old CloudFormation template which is using nodejs6 as it's runtime.

Creating an issue to track fixing this.

Fastly Log Processor Function optional in CFN template

We intend to support multiple CDN log formats with this project. Typical use-case should be that each CDN is enabled by a parameter in the CFN template.

CloudFront will be on by default, as we can control that set up in the template, but if we ever break cloudfront into an optional component, we can revisit this.

First step will be to create a toggle for the fast.ly support which will selectively deploy the lambda function and the updated bucket policy for S3. This will be False by default.

unable to deploy cloudformation template (deployment.yaml)

Hi, i have issues in deploying the cloudformation template. I follow your guide to deploy deployment.yaml but found errors.
Seems like it failed to create Schema (line 970-975) and the error message was the S3 Bucket not found.
Can you help to advise how can i solve this issue?

Update nodejs functions to 12.x

nodejs 8.x and 10.x are ending support. We need to update the lambda functions to 12.x to keep the template functional.

Synchronize documentation with published lab guides

The .md guides in this repo are out of sync with the published lab guides used at public events. The published guides have not been generated from markdown, so resyncing this will require some effort.

Published web guides:

1. self-deployed: https://aws-streaming-media-analytics-workshop.s3.amazonaws.com/index.html
2. deployed via AWS Event Engine: https://aws-streaming-media-analytics-workshop.s3.amazonaws.com/ee/index.html

Creating this issue with a goal of two outcomes:

1. included .md lab guides should 100% match the published web guides
2. a method for generating the published web guides should be included so we don't get out of sync again.

Customers would want a new UX for this workshop

Customers told me is difficult to follow this workshop with a laptop, because there is a lot of text and they need to scroll a lot to find the information.

I have showed them this theme used by eksworkshop.com.
https://learn.netlify.com/en/
All told me it will be easier to follow this workshop with this theme and "Automatic next/prev buttons to navigate through menu entries"

Runtime ImportModuleErrror for DeployFunction Lambda

{
"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module 'aws-sdk'\nRequire stack:\n- /var/task/index.js\n- /var/runtime/index.mjs",
"stack": [
"Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'",
"Require stack:",
"- /var/task/index.js",
"- /var/runtime/index.mjs",
" at _loadUserApp (file:///var/runtime/index.mjs:997:17)",
" at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1029:21)",
" at async start (file:///var/runtime/index.mjs:1192:23)",
" at async file:///var/runtime/index.mjs:1198:1"
]
}

S3 Bucket ACLs being disabled by default (new feature) cause deployment failure

Need to update the S3 buckets in the CloudFormation template to keep using ACLs in order to support CloudFront log delivery:

OwnershipControls:
Rules:
- ObjectOwnership: BucketOwnerPreferred

ref: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html#AccessLogsBucketAndFileOwnership

nodejs8.10 no longer supported in lambda

lambda has discontinued support for deploying functions with the nodejs8.10 runtime.

Issue to track migrating to a newer runtime.

Update isomorphic-fetch / node-fetch

node-fetch is a dependency of isomorphic-fetch which is used by activeuser-appsync-function. A minor security vuln has been reported in node-fetch and this package needs to be updated.

video playback failing due to CORS permissions in Firefox 71

Playing back video in Firefox 71 on Mac results in errors related to permissions (CORS?):

NotAllowedError: The play method is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://dl0q54uth5jdv.cloudfront.netsample-videos/hls/BigBuckBunny.m3u8. (Reason: CORS request did not succeed).

Add support for additional regions

The Streaming Media Analytics Workshop currently only supports:

• us-east-1
• us-west-2

Since developing the workshop, additional regions have added support for the variety of services used in this project.

Feature request to add support for additional regions:

• ap-southeast-1
• ap-southeast-2
• ap-northeast-1
• ap-northeast-2
• ap-south-1
• ca-central-1
• eu-west-1
• eu-west-2
• eu-west-3
• eu-north-1
• sa-east-1
• us-east-2

Update python functions to 3.x

Support for python2.7 is ending in Lambda. We need to update the python lambda functions to a 3.x runtime so the template remains deployable.