bensoer / cdk-photo-archive

CDK based AWS Infrastructure for Photo Archiving

👋 Hi, I’m @bensoer . A DevOps Engineer, Photographer, Outdoor enthusiast, and part-time philosopher. I love building things that help make these things in my world easier and more fun!

👀 Currently I'm exploring AWS Technologies, specifically the AWS CDK, TypeScript, Custom Resources and a bunch in between. I love automating things

🌱 I’m currently learning AWS Certifications, a bit of Terraform, TypeScript and the AWS CDK

💞️ I’m looking to collaborate on meaningful projects to make our real-world lives easier, as well improve functionality of existing toolings and frameworks that make our development processes smoother

cdk-photo-archive's Issues

Add CfnParameters and passing parameters to NestedStacks

Background
Currently all settings and configuration is done in CDK code. This means its done at synthesis, producing no parameter values for the generated CloudFormation code.

Problem
In order to integrate with the AWS Marketplace or increase further integrations in more ways. Settings at synthesis is not enough

Resolution
Implement CfnParameters at the root cdk-photo-archive stack. To work with the existing parameters set at synthesis time, make the default values for all CfnParameters to that of the configured values.

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.CfnParameter.html
https://bobbyhadz.com/blog/aws-cdk-parameters-example

Implementing this may also require different handling of conditional logic which now needs to be pushed to at runtime rather then syntehsis time: https://loige.co/create-resources-conditionally-with-cdk/

Configure also NestedStack parameters to pass around the needed parameters from there too: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.NestedStack.html#parameters

Tiering Storage Configuration Improvements

Background
Currently storage uses 3 tiers, Standard, Infrequent Access and Glacier. This is all set using the switchToWhateverTierAfterDays parameters.

Tiering is also assumed to be from most expensive to cheapest: Standard > Infrequent Access > Glacier . This functionality is assumed when viewing the settings

Problem
When switchToWhateverTierAfterDays is set to 0. The behavior is that the entry is not added to the lifecycle. This works if you want to skip InfrequentAccess and go straight to Glacier after X number of days. But if both settings are 0. Then everything will stay in Standard Tier. This is confusing and does not reflect intuitively to the expected functionality

Code Sample of the current functionality

if(switchToInfrequentAccessTierAfterDays !== undefined 
    && switchToInfrequentAccessTierAfterDays > 0){

    transitions.push(
        {
            storageClass: s3.StorageClass.INFREQUENT_ACCESS,
            transitionAfter: Duration.days(switchToInfrequentAccessTierAfterDays)
        }
    )
}else if(switchToInfrequentAccessTierAfterDays === undefined){
    transitions.push(
        {
            storageClass: s3.StorageClass.INFREQUENT_ACCESS,
            transitionAfter: defaultInfrequentAccessTransitionDuration
        }
    )
}

Resolution
Having 0 days should mean that it is then immediatly sent to the next storage tier. If all parameters have 0 days, then everything should be going straight to Glacier storage.

IF switchToInfrequentAccessTierAfterDays is 0. Then any upload should immediatly go into InfrequentAccess
IF switchToGlacierAccessTierAfterDays is 0. Then any upload should immediatly go into Glacier

IF switchToInfrequentAccessTierAfterDays is 0 AND switchToGlacierAccessTierAfterDays is 0. Then any upload should immediatly go info Glacier

Setup Lambda Versions

Background
cdk-photo-archive uses a number of lambdas for processing things. When updating these lambdas though, can cause wierd or unwanted states on the end user

Problem
Sometimes lambdas are in wierd states where they can't be fully deleted or the user needs to fully uninstall the cdk-photo-archive just to upgrade. This is not ideal

Solution
Implement blue/green deploys and use AWS Lambda Versions from the CDK and Cloudformation to deploy updates to lambdas within the cdk-photo-archive. Most importantly is the feature lambdas should be having this treatement on updates. Second place is the other processing and queueing lambdas

Create an API Endpoint for Accessing DynamoDB Records

Background
DynamoDB records can be enabled with a setting within the cdk-phot-archive. But they now can't be accessed in any meaningful way other then the AWS UI

Problem
DynamoDB records can be enabled with a setting within the cdk-phot-archive. But they now can't be accessed in any meaningful way other then the AWS UI

Solution
Create an API Gateway with it to access the information. Add also a setting to true/false enable/disable this featue

Create a Reactjs App to view dynamoDB table information

Background
A prerequisiste to this is the completion of #7

Problem
Now access this API information in a user friendly way. Not all users will be technical.

Solution
Create a Reactjs app that queries the API for the dynamoDB statistical information.

Some additional features that would be good in this UI:

• Authentication handled with AWS Cognito ?
• Access also the S3 data so that information can be cross referenced as to what has metric information and what does not. Essentially this UI functions like an S3 browser, just focused on photo buckets and artifacts

Switch Processing Lambdas over to Step Functions

Background
Current implementation has a number of lambdas calling eachother and passing data between eachother in a static format.

Problem
This can be massively simplified in code and CDK by using Step Functions

Solution
Use step functions for image processing from event all the way to feature handling