Only initial lambda version (1) works about cloudfront-authorization-at-edge

[ottokruse]

Hi Jesse. I just updated the code and added a "Version" parameter to the CloudFormation (SAM) template. While developing you can use that to force a new (valid) deployment of Lambda@Edge code. This worked for me, can you check if it also works for your workflow?

You can just put any value in the "Version" parameter; every time you want to redeploy the same stack, use a new version. If CloudFormation detects the version to have another value than during the previous stack deployment, it will update the Lambda@Edge functions. This is a bit of a "trick" to help CloudFormation understand something's changed.

[jesseadams]

To be clear, the versioning was working previously. Each time I changed the HTTP headers, for instance, the lambdas would update and the outputs would show the new version. For some reason, however, after my consuming stack updated the CloudFront behaviors with the new versions the lambdas no longer run. I thought my change was bad, but after reverting it and deploying a new version it was still not working. If I make my changes, drop the stack, and redeploy my changes work fine. I am just trying to avoid having to drop my consuming stack, this stack, and redeploying both every time I need to tweak one of the lambdas. It should be rare, I just don't want it to bite me in the future

Does that make sense?

[ottokruse]

Not following your workflow entirely but I suspect your issue will be solved now by the change I mentioned. Can you maybe try deploying again?

sam deploy --template-file packaged.yaml \
           --stack-name your-stack-name  \
           --capabilities CAPABILITY_IAM \
           --parameter-overrides [email protected] Version=$(date +%s) \
           --region us-east-1

Note the Version parameter. Which in this example is filled with a timestamp, which makes it different each time, thereby triggering the complete redeployment.

[ottokruse]

"For some reason, however, after my consuming stack updated the CloudFront behaviors with the new versions the lambdas no longer run."

Can you explain a bit more about the consuming stack? How are the stacks related?

[jesseadams]

So I had a pre-existing CFN stack that had CloudFront setup with a simple HTTP BasicAuth Lambda. I am migrating to this solution to see how it all works.

I added this parameter to my stack:

  AuthAtEdgeStackName:
    Type: String
    Description: The name of the AWS SAM stack that includes the lambdas and cognito setup for auth
    Default: cloudfront-authorization-at-edge

And here is a sample of how I am consuming the lambdas:

        DefaultCacheBehavior:
          Compress: true
          ForwardedValues:
            QueryString: true
          LambdaFunctionAssociations:
            - EventType: viewer-request
              LambdaFunctionARN:
                Fn::ImportValue:
                  !Sub "${AuthAtEdgeStackName}-CheckAuthHandler"
            - EventType: origin-response
              LambdaFunctionARN:
                Fn::ImportValue:
                  !Sub "${AuthAtEdgeStackName}-HttpHeadersHandler"
          TargetOriginId: s3Origin
          ViewerProtocolPolicy: redirect-to-https

I am deleting both stacks now and I will rebase your changes and let you know how it goes. If I run into further issues, I'll pass along more details on error messages, etc.

[jesseadams]

Upon redeploying, I see the same error I was previously seeing, but after the initial deploy.

503 ERROR
The request could not be satisfied.
The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
Generated by cloudfront (CloudFront)
Request ID: 5j-PiKXdNHNPgPDNJ46GcmPQvR0qm2jPh7VJH9ImcwnOmAe8o6kgPg==

[jesseadams]

The output shows version 2 for the lambdas arn:aws:lambda:us-east-1:**********:function:cloudfront-authorization-at-edge-ParseAuthHandler-1L6V711M9NOB7:2

[jesseadams]

Manually updating the default behavior lambdas to use version 1 resolves it.

[ottokruse]

It would be great if we can get your scenario to work. Not sure yet what the root cause might be, but 503 may mean the Lambda's are not returning a proper response to CloudFront. Can you take a look at their logs, and check that there's no exceptions being thrown?

The easiest way to find the log (in the right region) is through the CloudFront monitoring dashboard--in case you didn't know.

[ottokruse]

BTW this example might be of interest to you: reuse-auth-only.yaml

[jesseadams]

@ottokruse When I previously looked for debug I didn't see any log streams in CloudWatch nor did I see invocations in the monitoring of the lambdas.

This new example is nice. My only custom needs would be to have the following be a parameter that defaults to true to preserve your original use case on the UserPool

      AdminCreateUserConfig:
        AllowAdminCreateUserOnly: false

As well as an optional parameter for the email domain that gets used in a PreSignUp LambdaConfig for the UserPool with the following lambda in src/cognito/verify-email

exports.handler = (event, context, callback) => {
    console.log(event);

    // Split the email address so we can compare domains
    var address = event.request.userAttributes.email.split("@");

    if ( "example.com" === address[1]) {
        callback(null, event);
    } else {
        var error = new Error("Only authorized email addresses may register.");
        callback(error, event);
    }
};

This would enable a optional parameter set to let users self sign up if they are using an email address with the whitelisted domain. Finally, another optional parameter would be

      AutoVerifiedAttributes:
        - email

Which would require individuals that sign up to verify their email address after signing up.

[ottokruse]

Uncaught exceptions should be visible in the logs. For example, if the Lambda can't read it's config that would show in the logs. Can you check? And if you wanna be sure that the Lambda's are returning the right response format to CloudFront, maybe add a console.log to show what is returned too.

Thanks for that info on your use case. A solution might also be to be able to provide a user pool id as parameter. That would allow you to use your own user pool, with your custom config. What do you think?

[jesseadams]

This time I just deployed your solution without my stack to a new account. I have the same behavior. After I update the cloudformation stack for the first time (literally a noop, just a version bump) I start getting the 503 errors. The only CloudWatch Log Groups I see are the CFN custom resource handlers.

Manually updating the CloudFront Distro to use version 1 of the lambdas still fixes everything.

[ottokruse]

I will try to reproduce on my side. Can you paste me the exact steps you go through?

[jesseadams]

sam build --use-container
sam package --output-template-file packaged.yaml --s3-bucket example-bucket-name --region us-east-1
sam deploy --template-file packaged.yaml --stack-name cloudfront-authorization-at-edge --capabilities CAPABILITY_IAM --parameter-overrides AlternateDomainNames=example.org CreateCloudFrontDistribution=true Version=$(date +%s) --region us-east-1

Test and it works

sam deploy --template-file packaged.yaml --stack-name cloudfront-authorization-at-edge --capabilities CAPABILITY_IAM --parameter-overrides AlternateDomainNames=example.org CreateCloudFrontDistribution=true Version=$(date +%s) --region us-east-1

Test and it no longer works

[ottokruse]

Hi! I was able to reproduce your issue and find out the cause. There was an issue with the node dependency adm-zip that is used for adding configuration to the Lambda functions. I've changed the way that lib is used, to work around the issue, and redeployments work now.

[siviae]

@ottokruse I am sorry for, maybe, stupid question (I just started learning cloudformation and SAM), but I don't understand: why to pass configuration to edge lambdas as a json, requiring wrapper lambda for repackaging, when you could simply pass it using Environment.Variables property? Thanks in advance for response.

[ottokruse]

No worries. Lambda@Edge doesn't support Env Variables. This is explained in the blog (under heading "Using Lambda@Edge functions")