melodyyangaws / jam-challenge-spark-on-eks

AWS Jam Spark on EKS Made Easy

A python CDK v2 project for AWS JAM - Spark on EKS made easy.

Generate the CFN

export BUCKET_NAME_PREFIX=aws-jam-challenge-resources
export SOLUTION_NAME=spark-on-eks-made-easy

# go to the project directory
cd jam-challenge-spark-on-eks
./deployment/build-s3-dist.sh $BUCKET_NAME_PREFIX $SOLUTION_NAME

########################################################
# NOTE: convert json templates to yaml file on CFN designer console, then rename them to xxx.yaml, finally sync yaml templates to JAM S3 bucket. 
cd deployment/global-s3-assets/
aws s3 ls s3://aws-jam-challenge-resources/spark-on-eks-made-easy/
aws s3 sync . s3://aws-jam-challenge-resources/spark-on-eks-made-easy
########################################################
# sync regional lambda functions generated by CDK
cd ../regional-s3-assets
aws s3 sync . s3://aws-jam-challenge-resources/spark-on-eks-made-easy

# Sync the rest of application assets
cd ../../jam_asset
aws s3 sync ../source/app_resources/jupyter-values.yaml s3://aws-jam-challenge-resources/spark-on-eks-made-easy
aws s3 sync ../deployment/post-deployment.sh s3://aws-jam-challenge-resources/spark-on-eks-made-easy
aws s3 sync ../deployment/app_code/job/NYCTaxiCount.py s3://aws-jam-challenge-resources/spark-on-eks-made-easy

Clean up

Run the clean-up script with your CloudFormation stack name.The default name is SparkOnEKS.

cd aws-jam-spark-on-eks-made-easy
./deployment/delete_all.sh

Go to CloudFormation console, manually delete the remaining resources if needed.