Stochastic Decryption Time about aws-encryption-sdk-python

[timnlupo]

Update- I found that one of the keys in key_arns did not have a replica in my region. Adding a replica key in my region seems to resolve the issue.

However, I'm curious why just having the key in the key_ids stochastically makes it change speed?

[timnlupo]

Also, it seems that the first decrypt operation is always slower.

TIME ELAPSED:decrypting: 0.0799s
TIME ELAPSED:decrypting: 0.0263s
TIME ELAPSED:decrypting: 0.0277s

Is there some overhead that causes this? Is it possible to preload to avoid?

[alex-chew]

Hi @timnlupo, thanks for reaching out. Given the same ciphertext, consecutive decryptions should run deterministically. So any timing differences are likely to be caused by either network conditions between your host and AWS, or timing differences on the KMS side.

It's a bit hard to comment any more specifically on timing differences without more details. Can you please answer the following?

1. How many keys are in key_arns? In which regions do they reside? Are they multi-Region AWS KMS keys?
2. Where is this benchmark running? Is it running within the same region during each run?
3. Can you clarify what you mean by "the faster time" and "the slower time"? Are these timing distributions representative of more runs of your benchmark?

[timnlupo]

Hey @alex-chew. I figured out one of the issues- turns out I made a mistake configuring environment variables and the ciphertext that I was decrypting with the "slower time" (the second log I linked) was slower because it was encrypted with keys from two different regions. When I re-encrypted the content with two keys that were both in the same region, I could consistently reproduce the "faster time". That problem is solved

However, I noticed that on every run I see a pattern similar to the one below.

TIME ELAPSED:encrypting: 0.3682s
TIME ELAPSED:decrypting: 0.0799s
TIME ELAPSED:decrypting: 0.0263s
TIME ELAPSED:decrypting: 0.0277s
TIME ELAPSED:decrypting: 0.0263s
TIME ELAPSED:decrypting: 0.0277s
TIME ELAPSED:decrypting: 0.0269s
TIME ELAPSED:decrypting: 0.0253s
TIME ELAPSED:decrypting: 0.0234s
TIME ELAPSED:decrypting: 0.0262s
TIME ELAPSED:decrypting: 0.0252s
TIME ELAPSED:decrypting: 0.0287s
TIME ELAPSED:decrypting: 0.0551s
TIME ELAPSED:decrypting: 0.0279s
TIME ELAPSED:decrypting: 0.0284s
TIME ELAPSED:decrypting: 0.0313s
TIME ELAPSED:decrypting: 0.0300s
TIME ELAPSED:decrypting: 0.0284s
TIME ELAPSED:decrypting: 0.0277s
TIME ELAPSED:decrypting: 0.0288s
TIME ELAPSED:decrypting: 0.0271s

I notice the first decrypt (and sometimes other decrypts, eg the 0.0551s one) take ~3x slower then the fastest. Could a discrepancy this large be explained by the differences in ciphertext?

Not a big deal but I'm curious. Thanks so much.

[alex-chew]

Hi @timnlupo, glad to hear you were able to figure out the major timing difference.

Could a discrepancy this large be explained by the differences in ciphertext?

Assuming this most recent log is produced by running your original code, then there is no difference in ciphertext because the code decrypts the same ciphertext twenty times. So no significant discrepancy should be expected in the code running on your machine. Instead, the timing difference is more likely to be due to caching, either on the KMS side or perhaps in network routing - but it's hard to say conclusively without more details and diagnostic information.

[alex-chew]

I'm going to go ahead and close this out, but feel free to re-open it or create a new issue if you have any other questions or concerns.