mozilla-services / remote-settings-lambdas

Remote Settings scheduled tasks

Remote Settings Lambdas

A collection of scripts related to the Remote Settings service.

Sentry

All commands use Sentry to report any unexpected errors. Sentry can be configured with these environment variables, which are recommended, but not required:

• SENTRY_DSN: The DSN from the "Client Keys" section in the project settings in Sentry.
• SENTRY_ENV: The environment to use for Sentry, e.g. dev, stage or prod.

Commands

Each command can be run, either with Python:

$ python aws_lambda.py validate_signature

or via the Docker container:

$ docker run remote-settings-lambdas validate_signature

refresh_signature

Environment config:

• SERVER: server URL (default: http://localhost:8888/v1)
• REFRESH_SIGNATURE_AUTH: credentials, either user:pass or {access-token} (default: None)
• REQUESTS_TIMEOUT_SECONDS: Connection/Read timeout in seconds (default: 2)
• REQUESTS_NB_RETRIES: Number of retries before failing (default: 4)
• MAX_SIGNATURE_AGE: Refresh signatures that are older that this age in days (default: 7)

Note: In order to force refresh of all signatures, set MAX_SIGNATURE_AGE=0

Example:

$ REFRESH_SIGNATURE_AUTH=reviewer:pass  python aws_lambda.py refresh_signature

Looking at /buckets/monitor/collections/changes:
Looking at /buckets/source/collections/source: to-review at 2018-03-05 13:56:08 UTC ( 1520258168885 )
Looking at /buckets/staging/collections/addons: Trigger new signature: signed at 2018-03-05 13:57:31 UTC ( 1520258251343 )
Looking at /buckets/staging/collections/certificates: Trigger new signature: signed at 2018-03-05 13:57:31 UTC ( 1520258251441 )
Looking at /buckets/staging/collections/plugins: Trigger new signature: signed at 2018-03-05 13:57:31 UTC ( 1520258251547 )
Looking at /buckets/staging/collections/gfx: Trigger new signature: signed at 2018-03-05 13:57:31 UTC ( 1520258251640 )

backport_records

Backport the changes from one collection to another. This is useful if the new collection (source) has become the source of truth, but there are still clients pulling data from the old collection (destination).

Note: This lambda is not safe if other users can interact with the destination collection.

Environment config:

• SERVER: server URL (default: http://localhost:8888/v1)
• BACKPORT_RECORDS_SOURCE_AUTH: authentication for source collection
• BACKPORT_RECORDS_DEST_AUTH: authentication for destination collection (default: same as source)
• BACKPORT_RECORDS_SOURCE_BUCKET: bucket id to read records from
• BACKPORT_RECORDS_SOURCE_COLLECTION: collection id to read records from
• BACKPORT_RECORDS_SOURCE_FILTERS: optional filters when backporting records as JSON format (default: none, eg. "{"min_age": 42}")
• BACKPORT_RECORDS_DEST_BUCKET: bucket id to copy records to (default: same as source bucket)
• BACKPORT_RECORDS_DEST_COLLECTION:collection id to copy records to (default: same as source collection)
• REQUESTS_TIMEOUT_SECONDS: Connection/Read timeout in seconds (default: 2)
• REQUESTS_NB_RETRIES: Number of retries before failing (default: 4)
• SAFE_HEADERS: Add concurrency control headers to update requests (default: false)

Example:

$ BACKPORT_RECORDS_SOURCE_AUTH=user:pass BACKPORT_RECORDS_SOURCE_BUCKET=blocklists BACKPORT_RECORDS_SOURCE_COLLECTION=certificates BACKPORT_RECORDS_DEST_BUCKET=security-state BACKPORT_RECORDS_DEST_COLLECTION=onecrl  python3 aws_lambda.py backport_records

Batch #0: PUT /buckets/security-state/collections/onecrl/records/003234b2-f425-eae6-9596-040747dab2b9 - 201
Batch #1: PUT /buckets/security-state/collections/onecrl/records/00ac492e-04f7-ee6d-5fd2-bb12b97a4b7f - 201
Batch #2: DELETE /buckets/security-state/collections/onecrl/records/23 - 200
Done. 3 changes applied.

$ BACKPORT_RECORDS_SOURCE_AUTH=user:pass BACKPORT_RECORDS_SOURCE_BUCKET=blocklists BACKPORT_RECORDS_SOURCE_COLLECTION=certificates BACKPORT_RECORDS_DEST_BUCKET=security-state BACKPORT_RECORDS_DEST_COLLECTION=onecrl  python3 aws_lambda.py backport_records

Records are in sync. Nothing to do.

sync_megaphone

Send the current version of Remote Settings data to the Push server.

Does nothing if versions are in sync.

Environment config:

• SERVER: Remote Settings server URL (default: http://localhost:8888/v1)
• MEGAPHONE_URL: Megaphone service URL
• MEGAPHONE_READER_AUTH: Bearer token for Megaphone read access
• MEGAPHONE_BROADCASTER_AUTH: Bearer token for Megaphone broadcaster access
• BROADCASTER_ID: Push broadcaster ID (default: remote-settings)
• CHANNEL_ID: Push channel ID (default: monitor_changes)

Example:

$ SERVER=https://settings.prod.mozaws.net/v1 MEGAPHONE_URL="https://push.services.mozilla.com/v1" MEGAPHONE_READER_AUTH="a-b-c" MEGAPHONE_BROADCASTER_AUTH="d-e-f" python aws_lambda.py sync_megaphone

Test locally

$ make test

$ SERVER=https://firefox.settings.services.mozilla.com/v1/  python aws_lambda.py validate_signature

Local Kinto server

Best way to obtain a local setup that looks like a writable Remote Settings instance is to follow this tutorial

It is possible to initialize the server with some fake data, like for the Kinto Dist smoke tests:

$ bash /path/to/kinto-dist/tests/smoke-test.sh

Releasing

1. Create a release on Github on https://github.com/mozilla-services/remote-settings-lambdas/releases/new
2. Create a new tag X.Y.Z (This tag will be created from the target when you publish this release.)
3. Generate release notes
4. Publish release

License

Apache 2.0

remote-settings-lambdas's Issues

Add lambda to synchronize contents of two collections

copy-records --source-bucket=blocklists --source-collection=certificates --dest-bucket=security-state --dest-collection=onecrl

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

Add a way to execute as failure

In order to be able to test the Sentry integration, we should have a way to simulate an error

Move all this to amo2kinto repo?

refresh_signature lambda tries to access a non-existing collection and fails with 403

On -stage, v4.0.2, the refresh_signature lambda fails with the following error:

START RequestId: d05fc3a6-5ed2-11e8-8a09-6373fe64fc8c Version: $LATEST
Looking at /buckets/monitor/collections/changes: 
Looking at /buckets/main-workspace/collections/test-leplatrem: GET /v1/buckets/main-workspace/collections/test-leplatrem - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}: KintoException
Traceback (most recent call last):
  File "/var/task/aws_lambda.py", line 194, in refresh_signature
    collection_metadata = client.get_collection()['data']
  File "/var/task/kinto_http/__init__.py", line 595, in get_collection
    resp, _ = self.session.request('get', endpoint)
  File "/var/task/kinto_http/session.py", line 123, in request
    raise exception
kinto_http.exceptions.KintoException: GET /v1/buckets/main-workspace/collections/test-leplatrem - 403 403 - {'code': 403, 'errno': 121, 'error': 'Forbidden', 'message': 'This user cannot access this resource.'}

END RequestId: d05fc3a6-5ed2-11e8-8a09-6373fe64fc8c
REPORT RequestId: d05fc3a6-5ed2-11e8-8a09-6373fe64fc8c	Duration: 714.73 ms	Billed Duration: 800 ms 	Memory Size: 128 MB	Max Memory Used: 56 MB	

However, the collection in the above error message doesn't exist. The collection exists in main and main-preview buckets, but not in main-workspace bucket.

$ curl -s --user "$userpass" https://kinto-writer.stage.mozaws.net/v1/buckets/main-workspace/collections/test-leplatrem | jq .
{
  "code": 404,
  "errno": 110,
  "error": "Not Found",
  "details": {
    "id": "test-leplatrem",
    "resource_name": "collection"
  }
}

Python 3.7

Not sure what the state of 3.7 support is inside the libraries we depend on. E.g. kinto-signer.

Whatever we do, we must remember that we have 2 different Python environments. The one inside Dockerfile and the one inside the .travis.yml file.

Share canonical_json as library

Especially if the Normandy people are going to use it. They've already discovered some bugs with it (mozilla/normandy#1210, and possibly another one @mythmon ?).

`refresh_signature` crash when source collection is missing

It is probably acceptable to crash out loud in that situation, but at least the job should continue refreshing the remaining collections before exiting with an error

Upgrade to Python 3.6

• Upgrade to kinto-http.py 9
• Upgrade to kinto 7

Automate the zip creation

Reading https://github.com/mozilla-services/amo2kinto-lambda/blob/master/docs/source/usage.rst

I think it would be easy to automate the creation of the zip by running the build on the right AMI. for example ami-f0091d91

e.g. http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

The benefit is to make sure that the produced zip is fully compatible with amazon lambda

The remote-zip make target does not work anymore.

Travis CI free usage ends Dec 3; mozilla repos should switch to other CI platforms

We're opening this issue because your project has used Travis CI within the last 6 months. If you have already migrated off it, you can close and ignore this issue.

Travis CI is ending free builds on public repositories. travis-ci.com stopped providingthem in early November, and travis-ci.org will stop after December 31, 2020. To avoid disruptions to your workflows, you must migrate to another CI service.

For production use cases, we recommend switching to CircleCI. This service is already widely used within Mozilla. There is a guide to migrating from Travis CI to CircleCI available here.

For non production use cases, we recommend either CircleCI or Github Actions. There is a guide to migrating from Travis CI to Github Actions available here. Github Actions usage within Mozilla is new, and you will have to work with our github administrators to enable specific actions following this process.

If you have any questions, reach out in #github-admin:mozilla.org on matrix.

Add open source software license

This Mozilla repository has been identified as lacking a license. Consistent with Mozilla's Licensing Policy an open source license should be applied to the code in this repository.

Please add an appropriate LICENSE.md file to the root directory of the project. In general, Mozilla's licensing policies are as follows:

• Client-side products created by Mozilla employees or contributors should use the Mozilla Public License, Version 2.0 (MPL).

• Server-side products or utilities that support Mozilla products may use either the MPL or the Apache License 2.0 (Apache 2.0).

In special cases, another license might be appropriate. If the repository is a fork of another repository it must apply the license of the original. Similarly, another license might be appropriate to match that of a broader project (for example Rust crates that Firefox depends on are published under an Apache 2.0 / MIT dual license, as that is the dual license used by the Rust programming language and projects).

Please ensure that any license added to the LICENSE.md file matches other licensing information in the repository (for example, it should match any license indicated in a setup.py or package.json file).

Mozilla staff can access more information in our Software Licensing Runbook – search for “Licensing Runbook” in Confluence to find it.

If you have any questions you can contact Daniel Nazer who can be reached at dnazer on Mozilla email or Slack.

OPENLIC-2023-01

blockpages_generator lambda sporadic failure due to "No space left on device" error

We got paged from time to time due to blockpages_generator lambda failures.

Upon checking the logs, it was caused by errors similar to below

[Errno 28] No space left on device: '/tmp/tmp7cjkc3ey': OSError
Traceback (most recent call last):
File "/var/task/aws_lambda.py", line 288, in blockpages_generator
target_dir = mkdtemp()
File "/var/lang/lib/python3.6/tempfile.py", line 368, in mkdtemp
_os.mkdir(file, 0o700)
OSError: [Errno 28] No space left on device: '/tmp/tmp7cjkc3ey'

The failure is sporadic and it usually doesn't happen very often, but for the recent 4 days, it has happened twice.

Is it possible to either clean the temporary space (before or after work), or catch the exception and ignore it?

This will help reduce false alerts, besides the lambda usually correct itself after a few re-runs, so it's basically when this happens, it's not actionable by ops.

refresh_signature should continue on 403 errors

When a 403 error occurs on a collection, the iteration should continue and process the other collections.
The collected errors can then be thrown at the end of the iteration

Move to CircleCI

Cuz it's all the rage in Mozilla Services.

Strip lambdas archive content...

We are close to the limitation of 20Mo...

It's mainly botocore...

$ du -hs * | sort -rh 
89M	total
47M	botocore
14M	lxml
8,7M	cryptography
3,8M	docutils
2,2M	unidecode
1,5M	boto3
1,1M	amo2kinto
1016K	pycparser
940K	jinja2

I think we could strip some of the folders in the botocore/data package

 du -hs botocore/data/* | sort -rh
45M	total
7,0M	botocore/data/ec2
3,5M	botocore/data/cloudfront
1,2M	botocore/data/rds
708K	botocore/data/sagemaker
672K	botocore/data/elasticache
660K	botocore/data/pinpoint

Change sentry_sdk integration when no longer AWS Lambda

As discussed, in the PR adding sentry_sdk, will use the AwsLambdaIntegration integration which is probably right when you use AWS Lambda. But if we no longer execute this whole script in AWS Lambda we'll want to drop that integration.

Note-to-self; Here are the docs about using sentry_sdk in other serverless environments: https://docs.sentry.io/platforms/python/serverless/

Unclear errors messages

See mozilla-services/amo2kinto#83

Related #27

Use x5u instead of public_key to validate signatures

validate_signature does not work on read-only instance since #29

Since #29 we lookup the signer capability to validate the signature.

This does not work on the public/readonly instance https://firefox.settings.services.mozilla.com/v1/ since the kinto-signer public is not enabled there :/

SERVER=https://firefox.settings.services.mozilla.com/v1/ python aws_lambda.py validate_signature
Looking at /buckets/monitor/collections/changes: 
Traceback (most recent call last):
  File "aws_lambda.py", line 330, in <module>
    function(event, context)
  File "aws_lambda.py", line 48, in validate_signature
    source = get_signed_source(server_info, collection)
  File "aws_lambda.py", line 142, in get_signed_source
    signed_resources = server_info['capabilities']['signer']['resources']
KeyError: 'signer'

Move schema.json from amo-blocklist-ui repo to cloudops

Like others... https://github.com/mozilla-services/cloudops-deployment/blob/761e4fbc63da11778347187ce92c24c4520000b0/projects/kinto/puppet/yaml/permissions/kinto.prod.yaml#L114

Approving changes does NOT fully invalidate Cloudfront cache

This issue is similar to #46 in the sense that validate_signature lambda keeps failing until we manually run refresh_signature lambda.

What happened was at 8:55am PDT someone approved some changes on staging/certificates collection, and afterwards the validate_signature lambda started to fail and paged operations. We had to log in and manually run refresh_signature lambda to correct it.

The validate_signature lambda error looks like this:

Signature verification failed on /buckets/blocklists/collections/certificates
- Signed on: 1544806548885 (2018-12-14 16:55:48 UTC)
- Records timestamp: 1541184580803 (2018-11-02 18:49:40 UTC): ValidationError
Traceback (most recent call last):
File "/var/task/aws_lambda.py", line 139, in validate_signature
raise ValidationError("\n" + "\n\n".join(error_messages))
aws_lambda.ValidationError:
Signature verification failed on /buckets/blocklists/collections/certificates
- Signed on: 1544806548885 (2018-12-14 16:55:48 UTC)
- Records timestamp: 1541184580803 (2018-11-02 18:49:40 UTC)

Looking at kinto-writer's log, the application seemed to have tried to invalidate the cloudfront cache.

Dec 14 16:55:49 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544806549341082624, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 16:55:49 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544806549360110848, "Type": "kinto_megaphone.listeners", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Sending version: remote-settings, monitor_changes"}}
Dec 14 16:55:49 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544806549442699520, "Type": "kinto_megaphone.megaphone", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Sent version to megaphone. Response was 200"}}
Dec 14 16:55:52 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544806552562391296, "Type": "subrequest.summary", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 44, "Fields": {"agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0", "path": "/v1/buckets/staging/collections/certificates", "method": "GET", "lang": "en-US,en;q=0.5", "errno": 0, "batch_size": 3, "
Dec 14 16:55:52 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544806552618742528, "Type": "subrequest.summary", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 44, "Fields": {"agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0", "path": "/v1/buckets/blocklists-preview/collections/certificates", "method": "GET", "lang": "en-US,en;q=0.5", "errno": 0, "batch_
Dec 14 16:55:52 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544806552684661760, "Type": "subrequest.summary", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 44, "Fields": {"agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0", "path": "/v1/buckets/blocklists/collections/certificates", "method": "GET", "lang": "en-US,en;q=0.5", "errno": 0, "batch_size": 3

However, we suspect it didn't invalidate what needed to be invalidated because not only validate_signature lambda started to fail after 8:55am PDT, but in the Cloudfront invalidation logs, we only see one invalidation request, whereas when we manually triggered refresh_signature lambda, it triggered much more invalidation requests, like below,

Dec 14 17:53:58 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810038524194048, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 47, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:53:59 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810039192479744, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 47, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:53:59 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810039542577408, "Type": "kinto.core.storage.postgresql.pool.QueuePoolWithMaxBacklog", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Connection <connection object at 0x7fb211358e88; dsn: 'user=kinto password=xxx dbname=kinto host=kinto-rds-prod.ctwh8irpumru.us-west-2.rds.amazonaws.com port=5432', closed: 0> exceeded timeout; recycling"}}
Dec 14 17:54:00 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810040259059456, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:00 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810040953859584, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:01 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810041959233536, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 44, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:02 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810042443594496, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 44, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:03 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810043468941312, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:03 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810043936243968, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:05 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810045060901632, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:05 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810045717103360, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:06 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810046769856000, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 44, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:07 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810047303014656, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 44, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:08 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810048370542592, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 47, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:08 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810048814711296, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 47, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:09 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810049901712128, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 48, "Fields": {"msg": "Invalidated CloudFront cache at /v1/*"}}
Dec 14 17:54:10 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810050473050112, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:11 ip-172-31-10-135 docker-kinto[2604]: [busyness] worker nr 6 30s average busyness is at 8%
Dec 14 17:54:11 ip-172-31-10-135 docker-kinto[2604]: [busyness] worker nr 7 30s average busyness is at 8%
Dec 14 17:54:11 ip-172-31-10-135 docker-kinto[2604]: [busyness] worker nr 8 30s average busyness is at 17%
Dec 14 17:54:11 ip-172-31-10-135 docker-kinto[2604]: [busyness] 30s average busyness of 3 worker(s) is at 11%
Dec 14 17:54:11 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810051535116800, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:11 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810051976226048, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:13 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810053193980160, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:13 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810053642588160, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:14 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810054826879488, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 44, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:15 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810055273828608, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 44, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:16 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810056593531648, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:17 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810057052123648, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:18 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810058332531456, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:18 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810058893177856, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:20 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810060085889024, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:20 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810060551599104, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:21 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810061741596928, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:22 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810062208252416, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:23 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810063364969984, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:23 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810063773406720, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 47, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:24 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810064903221760, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:25 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810065337822720, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:26 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810066521855744, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:26 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810066923068928, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 48, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:29 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810069829135616, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 44, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}
Dec 14 17:54:30 ip-172-31-10-135 docker-kinto[2604]: {"Timestamp": 1544810070312627456, "Type": "kinto_signer.updater", "Logger": "kinto", "Hostname": "ip-172-31-10-135.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 3, "Pid": 44, "Fields": {"msg": "Cache invalidation request failed.", "error": "TooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')", "traceback": "Uncaught exception:\n  File \"/usr/local/lib/python3.7/site-packages/kinto_signer/updater.py\", line 405, in invalidate_cloudfront_cache\n    'CallerReference': '{}-{}'.format(timestamp, uuid.uuid4())\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 320, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/botocore/client.py\", line 623, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n<class 'botocore.errorfactory.TooManyInvalidationsInProgress'>\nTooManyInvalidationsInProgress('An error occurred (TooManyInvalidationsInProgress) when calling the CreateInvalidation operation: Processing your request will cause you to exceed the maximum number of in-progress wildcard invalidations.')\n"}}

Signature validation of empty collections

See https://bugzilla.mozilla.org/show_bug.cgi?id=1443031

• Signed empty collection cf. Kinto/kinto-signer#201
• New empty collection not signed at all (no signature attribute, should probably be fixed upstream cf. Kinto/kinto-signer#226)

Add certificate chain validation in the validating signature lambda.

Update title and readme to provide infos

What is this repo?

Who uses it?

What does this do?

`status` can be missing on new/unused collections

@autrilla reported this today:

https://irccloud.mozilla.com/pastebin/Uc7rzzBT/

I think some collections have no status, or worse yet, status: null, and I think we should handle this. At the very least, don't crash. Maybe better would be, for such collections, to check that the data in the corresponding destination collections is [].

Uptake health will get noisy

The uptake_health command sends a string to Sentry. That string contains not just the name but individual numbers. These numbers are about to change every 24h.

That means, that suppose we get an Sentry notification about the uptake health of main/foo-bar. Oh no! We investigate and worry and perhaps even change something in mozilla-central or in some configuration of some server. Either way, those changes will take some time to take effect fully. That means that for every single day thereafter we'll get new Sentry email notifications. Day after day.

The risk is that the Sentry notification traffic gets too noisy we become desensitized to it and start ignoring them. All of them!

Perhaps it's too early to worry but this issue is for discussing and potentially solving this issue.

Add more context on validation errors.

Add sentry on errors.

Allow loading *auth keys from Env variable exposed by CloudFormation and decoded by KMS

A new feature in AWS Lambda is the ability to load secrets from env variables.

See https://aws.amazon.com/blogs/aws/new-for-aws-lambda-environment-variables-and-serverless-application-model/

We want to support that for json2kinto editor_auth, reviewer_auth and auth

Decrease requests timeout and retry if necessary

The validate_signature function should take collections list from changes endpoint

Why would we want a different list of collections than the one used by Firefox?

It would avoid us doing https://github.com/mozilla-services/cloudops-deployment/pull/1580 (we want remote settings self-service in the end)

Different thresholds per status?

(GitHub doesn't allow transfer of issues across orgs so I manually copy-n-pasted this issue from: mozilla/remote-settings-uptake-health#18)

Currently we only define the error threshold percent globally or per source.

But in some situations we may want to have different thresholds per status.

For example, we could consider that 5% of network errors is acceptable, whereas 2% of signature errors is too high.

Download records in parallel

Currently, the validate signature lambda does everything sequentially and takes sometimes more than 5min to be executed

When running it locally (takes 1min on my machine BTW...) I notice that the downloading is what takes more time.

Failing backport records

We still receive 412 Modified Meanwhile errors on STAGE where version 5.3.3 is supposed to be deployed https://sentry.prod.mozaws.net/operations/kinto-lambdas-stage/issues/7699994/

I can't understand what's going on. I cannot reproduce it... For example with the dev server:

http PUT https://kinto.dev.mozaws.net/v1/buckets/test -a admin:s3cr3t
http PUT https://kinto.dev.mozaws.net/v1/buckets/test/collections/test -a admin:s3cr3t
http PUT https://kinto.dev.mozaws.net/v1/buckets/test/collections/test/records/haha -a admin:s3cr3t
http PUT https://kinto.dev.mozaws.net/v1/buckets/test2 -a admin:s3cr3t
http PUT https://kinto.dev.mozaws.net/v1/buckets/test2/collections/test2 -a admin:s3cr3t

Sync the 2 collections:

SERVER=https://kinto.dev.mozaws.net/v1 BACKPORT_RECORDS_SOURCE_AUTH=admin:s3cr3t BACKPORT_RECORDS_SOURCE_BUCKET=test BACKPORT_RECORDS_SOURCE_COLLECTION=test BACKPORT_RECORDS_DEST_BUCKET=test2 BACKPORT_RECORDS_DEST_COLLECTION=test2  python3 aws_lambda.py backport_records

Batch #0: PUT /buckets/test2/collections/test2/records/haha - 201 
Done. 1 changes applied.

Modify the record on source:

echo '{"data": {"pouet": 2}}' | http PUT https://kinto.dev.mozaws.net/v1/buckets/test/collections/test/records/haha -a admin:s3cr3t

Sync again:

SERVER=https://kinto.dev.mozaws.net/v1 BACKPORT_RECORDS_SOURCE_AUTH=admin:s3cr3t BACKPORT_RECORDS_SOURCE_BUCKET=test BACKPORT_RECORDS_SOURCE_COLLECTION=test BACKPORT_RECORDS_DEST_BUCKET=test2 BACKPORT_RECORDS_DEST_COLLECTION=test2  python3 aws_lambda.py backport_records
Batch #0: PUT /buckets/test2/collections/test2/records/haha - 200 
Done. 1 changes applied.

Cannot use multiprocessing on AWS lambda

[Errno 38] Function not implemented: OSError
Traceback (most recent call last):
File "/var/task/aws_lambda.py", line 76, in validate_signature
pool = multiprocessing.Pool(processes=3)
File "/var/lang/lib/python3.6/multiprocessing/context.py", line 119, in Pool
context=self.get_context())
File "/var/lang/lib/python3.6/multiprocessing/pool.py", line 150, in __init__
self._setup_queues()
File "/var/lang/lib/python3.6/multiprocessing/pool.py", line 243, in _setup_queues
self._inqueue = self._ctx.SimpleQueue()
File "/var/lang/lib/python3.6/multiprocessing/context.py", line 112, in SimpleQueue
return SimpleQueue(ctx=self.get_context())
File "/var/lang/lib/python3.6/multiprocessing/queues.py", line 324, in __init__
self._rlock = ctx.Lock()
File "/var/lang/lib/python3.6/multiprocessing/context.py", line 67, in Lock
return Lock(ctx=self.get_context())
File "/var/lang/lib/python3.6/multiprocessing/synchronize.py", line 163, in __init__
SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
File "/var/lang/lib/python3.6/multiprocessing/synchronize.py", line 60, in __init__
unlink_now)
OSError: [Errno 38] Function not implemented

https://aws.amazon.com/es/blogs/compute/parallel-processing-in-python-with-aws-lambda/

https://stackoverflow.com/a/48667988

Run lambdas from Docker container

In order to be able to run it on GCP

validate_signature lambda error due to stale cloudfront cache

Previously, we removed the invalidate_cache lambda, and started to rely on the code in kinto-signer to do the job (i.e. invalidating the cloudfront cache).

However, today we had an issue that the validate_signature lambda failed because the cache wasn't up-to-date. As soon as we manually invalidated the cloudfront cache, the error went away.

While checking the invalidation history of the cloudfront cache, we notice that it almost always happens at 19:41 UTC (see https://screenshots.firefox.com/f4I5iwggdLa5wdmU/console.aws.amazon.com). Is there a reason why it always happens at the same time, because as far as I understand the invalidation should happen whenever people sign something, thus it makes more sense if the time were random.

The reason why 19:41 UTC might be relevant is that the alert about the lambda error happened a few minutes before 19:41, so if kinto-signer saves the task of invalidating the cache in a queue, and only does it at 19:41 every day when the queue is not empty, then it would have caused an issue like this (i.e. it waited too long to invalidate the cache).

For reference, the lambda error we saw looks like this:

START RequestId: 474dc323-57ac-11e8-aef8-c361b15461c5 Version: $LATEST
Read collection list from /buckets/monitor/collections/changes
01/15 /buckets/blocklists-preview/collections/addons:  OK
02/15 /buckets/blocklists/collections/addons:  ⚠ BAD Signature ⚠
03/15 /buckets/blocklists/collections/certificates:  OK
04/15 /buckets/blocklists-preview/collections/certificates:  OK
05/15 /buckets/blocklists/collections/plugins:  OK
06/15 /buckets/blocklists-preview/collections/plugins:  OK
07/15 /buckets/security-state-preview/collections/intermediates:  SKIP
08/15 /buckets/security-state/collections/intermediates:  SKIP
09/15 /buckets/blocklists-preview/collections/gfx:  OK
10/15 /buckets/blocklists/collections/qa:  OK
11/15 /buckets/focus-preview/collections/experiments:  SKIP
12/15 /buckets/pinning-preview/collections/pins:  SKIP
13/15 /buckets/focus/collections/experiments:  SKIP
14/15 /buckets/pinning/collections/pins:  OK
15/15 /buckets/blocklists/collections/gfx:  OK

Signature verification failed on /buckets/blocklists/collections/addons
 - Signed on: 1526323825645 (2018-05-14 18:50:25 UTC)
 - Records timestamp: 1525377135149 (2018-05-03 19:52:15 UTC): ValidationError
Traceback (most recent call last):
  File "/var/task/aws_lambda.py", line 111, in validate_signature
    raise ValidationError("
" + "

".join(error_messages))
aws_lambda.ValidationError: 
Signature verification failed on /buckets/blocklists/collections/addons
 - Signed on: 1526323825645 (2018-05-14 18:50:25 UTC)
 - Records timestamp: 1525377135149 (2018-05-03 19:52:15 UTC)

END RequestId: 474dc323-57ac-11e8-aef8-c361b15461c5
REPORT RequestId: 474dc323-57ac-11e8-aef8-c361b15461c5	Duration: 70806.56 ms	Billed Duration: 70900 ms 	Memory Size: 128 MB	Max Memory Used: 83 MB	

Remove the kinto-signer dependency

canonical_json() is a 4 lines function

https://github.com/Kinto/kinto-signer/blob/11a3e1513ed0b180c303669e03a20fd95991e1fc/kinto_signer/serializer.py#L5-L11

and to verify the signature it could make sense to use a different code as the one used by the server (double-check).

And it would get rid of a huge (useless) dependency tree (kinto-signer > kinto > pyramid > ....)