GithubHelp home page GithubHelp logo

rungutan / sentry-fargate-cf-stack Goto Github PK

View Code? Open in Web Editor NEW
61.0 61.0 16.0 28.44 MB

AWS CloudFormation template to launch a highly-available Sentry 20 stack through ECS Fargate at the minimum cost possible

License: Apache License 2.0

Dockerfile 0.90% Shell 5.07% Python 15.31% Ruby 11.78% HTML 66.95%

sentry-fargate-cf-stack's People

Contributors

kiskoza avatar mariusmitrofan avatar nodomain avatar rgunning avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

m-bassam94 dgomesbr nodomain praveen-livspace fzp gaurav2github kiskoza rkhyd rgunning chrisl85 borghol noname1908 daunjung-dev dkgohel allanoricil ahmed3sam

sentry-fargate-cf-stack's Issues

can import template

Hello, when import cloudformation template file, I got these errors, pleas help me check.

The following resource types are not supported for resource import:

AWS::SecretsManager::Secret,AWS::SecretsManager::Secret,AWS::SecretsManager::Secret,AWS::EC2::VPCGatewayAttachment,AWS::ElasticLoadBalancingV2::TargetGroup,AWS::ElasticLoadBalancingV2::TargetGroup,AWS::EC2::Route,AWS::EC2::SecurityGroupIngress,AWS::EC2::SecurityGroupIngress,AWS::EC2::SecurityGroupIngress,AWS::EC2::NetworkAclEntry,AWS::EC2::NetworkAclEntry,AWS::EC2::NetworkAclEntry,AWS::EC2::NetworkAclEntry,AWS::EC2::SubnetNetworkAclAssociation,AWS::EC2::SubnetNetworkAclAssociation,AWS::RDS::DBSubnetGroup,AWS::EC2::SubnetNetworkAclAssociation,AWS::ElastiCache::SubnetGroup,AWS::EC2::SubnetNetworkAclAssociation,AWS::EC2::SubnetNetworkAclAssociation,AWS::EC2::SubnetRouteTableAssociation,AWS::EC2::SubnetRouteTableAssociation,AWS::EC2::SubnetRouteTableAssociation,AWS::EC2::SubnetNetworkAclAssociation,AWS::EC2::SecurityGroupIngress,AWS::EC2::SecurityGroupIngress,AWS::EC2::SubnetRouteTableAssociation,AWS::EC2::SubnetRouteTableAssociation,AWS::EC2::VPCEndpoint,AWS::EC2::SubnetRouteTableAssociation,AWS::OpsWorks::Stack,Custom::KafkaCreateConfiguration,AWS::ElastiCache::ReplicationGroup,AWS::EC2::Route,AWS::EC2::Route,AWS::EC2::Route,AWS::Route53::RecordSetGroup,AWS::Route53::RecordSetGroup,AWS::Route53::RecordSetGroup,AWS::ElasticLoadBalancingV2::TargetGroup,AWS::ElasticLoadBalancingV2::TargetGroup,AWS::Route53::RecordSetGroup,AWS::ElasticLoadBalancingV2::TargetGroup,AWS::OpsWorks::Layer,AWS::MSK::Cluster,AWS::ApplicationAutoScaling::ScalableTarget,AWS::OpsWorks::Instance,AWS::ApplicationAutoScaling::ScalableTarget,Custom::KafkaGetBrokers,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::OpsWorks::ElasticLoadBalancerAttachment,AWS::ApplicationAutoScaling::ScalingPolicy,Custom::ClickHouseBootstrapSingleExecution,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,AWS::ApplicationAutoScaling::ScalableTarget,Custom::DelayClickHouseBootstrap,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,AWS::ApplicationAutoScaling::ScalingPolicy,Custom::SentryMigrationSingleExecution,Custom::DelaySentryMigration,Custom::SnubaBootstrapSingleExecution,Custom::DelaySnubaBootstrap,Custom::SnubaMigrationsSingleExecution,Custom::DelaySnubaMigrations,Custom::SentryInitialUserSingleExecution

Trying to Integrate google auth

Trying to Integrate google auth but showing Authorization error
Error 400 : invalid_request
Missing required parameter: client_id

RFE: make ElastiCache and RDS highly available

While all of the components are set up in a fail safe, redundant way, both ElastiCache and RDS use only one instance.

Since these are important components in the stack’s architecture I propose to provide the option in the template to set them up redundantly, e.g a Redis Cluster and a PostgreSQL instance with fail over.
The default setup could still be “cheaper” and without redundancy.

Clickhouse just broke itself

Hey,

I found out that Clickhouse stopped working yesterday afternoon. For some reason the pidfile cannot be opened anymore since then.

Aug  5 22:15:57 sentry-clickhouse-1 clickhouse-server[4544]: Code: 76, e.displayText() = DB::ErrnoException: Cannot open file /run/clickhouse/clickhouse-server.pid, errno: 2, strerror: No such file or directory, Stack trace (when copying this message, always include the lines below):
Aug  5 22:15:57 sentry-clickhouse-1 clickhouse-server[4544]: 0. DB::throwFromErrnoWithPath(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) @ 0x7b5e959 in /usr/bin/clickhouse
Aug  5 22:15:57 sentry-clickhouse-1 clickhouse-server[4544]: 1. DB::StatusFile::StatusFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void (DB::WriteBuffer&)>) @ 0x7bceedc in /usr/bin/clickhouse
Aug  5 22:15:57 sentry-clickhouse-1 clickhouse-server[4544]: 2. void std::__1::__optional_storage_base<DB::StatusFile, false>::__construct<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void (DB::WriteBuffer&)>&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, std::__1::function<void (DB::WriteBuffer&)>&) @ 0x7c3c171 in /usr/bin/clickhouse

I found out that there is a systemd daemon and an old init.d file with different settings. Does OpsWorks update stuff? Did the mess up happen then? I did not touch my Sentry stack for a longer time.

I fixed it by manually setting the pidfile to /tmp/... in the /etc/systemd/system/clickhouse-server.service file:

[Unit]
Description=ClickHouse Server (analytic DBMS for big data)
Requires=network-online.target
After=network-online.target

[Service]
Type=simple
User=clickhouse
Group=clickhouse
Restart=always
RestartSec=30
RuntimeDirectory=clickhouse-server
ExecStart=/usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/tmp/clickhouse-server.pid
LimitCORE=infinity
LimitNOFILE=500000
LimitNPROC=40960
CapabilityBoundingSet=CAP_NET_ADMIN CAP_IPC_LOCK CAP_SYS_NICE

[Install]
WantedBy=multi-user.target

Perhaps you have an idea.

Cheers.

RFE: use autoscaling for workers

After some debugging my issue with Sentry being stuck processing events from a high traffic site where we use it for CSP reports and scaling Kafka, RDS and Redis during the process I found that the single worker was using 100% CPU. The template is not using autoscaling currently. What is the reason for this?
I propose the same scaling rules like for all other tasks. I am happy to send a pull request if you agree.
In my scenario I now temporarily scaled to 50 workers to let the queue be processed.

GeoIpUpdate does not seem to work with Sentry

In tag 1.12.* geoipupdate was introduced and database (*.mmdb) file was successfully pushed into Relay and Sentry Base containers.

However, weird errors popped up in both images and I was forced to "revert" the template and forward-release it with tag 1.13.0.

Log from GeoIpUpdate container:

Acquired lock file lock (/geoipupdate-mmdb/.geoipupdate.lock)
Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-City/update?db_md5=00000000000000000000000000000000
geoipupdate version 4.6.0
Using config file /etc/GeoIP.conf
Using database directory /geoipupdate-mmdb
Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-City
AccountID <<OBFUSCATED>>
LicenseKey <<OBFUSCATED>>
EditionIDs GeoLite2-City

Errors in Sentry base image:

sentry.utils.geo: Error opening GeoIP database in Rust: /geoip/GeoLite2-City.mmdb
sentry.utils.geo: Error opening GeoIP database: /geoip/GeoLite2-City.mmdb

Errors in Relay image:

2021-02-27T19:13:02Z [relay_log::utils] ERROR: configuration error
caused by: could not load the Geoip Db
caused by: IoError: entity not found

@nodomain Sorry mate. Tried. Had no luck. If you have any idea what's going on, you're more than welcome to give it a shot :)

PS: Some articles that should help but make no sense to me ->

getsentry/self-hosted#310
https://forum.sentry.io/t/how-to-set-geoip-path-mmdb/10198

Botocore deprecation notice of vendored requests library

AWS notice message:

We are contacting you regarding actions needed on your account(s) to prevent disruption to your CloudFormation stacks. Starting on April 01, 2021, AWS Lambda will no longer support the botocore.requests library in Python runtimes [1][2]. If you are using the cfn-response module for a Lambda-backed custom resource, you must modify your Lambda function resource’s Code or Runtime property and update your stack(s) in order to get the latest version of the cfn-response module which removed its dependency on botocore.requests. If you do not update your Lambda function by April 01, 2021, you will no longer be able to create, update or delete your custom resource. If you are updating the Runtime property, please include the lambda:UpdateFunctionCode action in your IAM policy.

If you are hosting your Lambda function code from an S3 bucket, you can refer to the updated cfn-response module’s source code in our documentation [3]. For more information on how to update lambda function can be found in knowledge center [4].

We need to force a code change in all Lambda functions by editing either the Code or the Runtime.

Based on AWS notes from here https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-cfn-response-lambda/, a simple comment addition is enough:

Note: For more information, see the code examples in the "Module source code" section of cfn-response module.

The cfn-response module code example uses botocore.requests in the Lambda function’s deployment package.

To update the cfn-response module to the latest version that uses urllib3, update the function’s inline code in the AWS CloudFormation template by adding a comment to the inline Lambda function's code. For example:
+            # This comment was added to force an update on this function's code
             responseData = {'Message': 'Hello {}!'.format(event['ResourceProperties']['Name'])}
             cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")

Update to 1.11.1 fails with Vendor response doesn't contain KafkaBroker1..3 key in object

On my staging environment everything went fine yesterday. On my production one it fails with these errors:
The only difference besides the instance type is that I manually extended the EBS storage on production already.

Please help - I already found that it seems to have something to do with the result of the lambda because it will output the keys only on "Create" but I am stuck. https://github.com/Rungutan/sentry-fargate-cf-stack/blob/main/cloudformation-template.yaml#L1828

CustomResource attribute error: Vendor response doesn't contain KafkaBroker3 key in object arn:aws:cloudformation:eu-central-1:722716701248:stack/sentry-prod/6e810c90-5cb1-11eb-8bbc-0207ca607726|KafkaGetBrokersLambdaResult|6c61345e-58f6-4b10-b64d-28e849cd4cdd in S3 bucket cloudformation-custom-resource-storage-eucentral1

image

Thanks

Updates

How are you handling updates for Sentry versions? Given we cannot actually REMOTE into the machines and upgrade them, as that's not how this whole process works. And, given that Sentry doesn't include an 'update' feature within it's own code; how does this stack handle updates?

o1.ingest.domain.tld is not created

Hi,

awesome project. I had to try it immediately.

Unfortunately, the DNS entry for oi1.ingest.domain.tld is missing in Route53. How to fix it?

Thanks
Fabian

Something went wrong during upgrade

After successfully upgrading the stage environment, the upgrade did not successfully upgrade the production environments. SnubaOutcomesConsumer exits with an error like this: getsentry/snuba#760

Looking at the thread it seems that the DB migration in Clickhouse did not work and I'd like to execute the commands that are shown in the thread but I don't know how.

Furthermore the UI also generates similar errors:

QueryMissingColumn: [47] DB::Exception: Missing columns: 'quantity' 'category' while processing query: 'SELECT outcome AS _snuba_outcome, category AS _snuba_category, project_id AS _snuba_project_id, sum(quantity) AS _snuba_quantity, _snuba_project_id, _snuba_outcome, _snuba_category, _snuba_quantity FROM outcomes_hourly_local PREWHERE _snuba_project_id IN (1, 2, 3, 4, 5, 7, 8, 9, 35) WHERE ((org_id AS _snuba_org_id) = 1) AND ((timestamp AS _snuba_timestamp) >= toDateTime('2021-04-16T00:00:00', 'Universal')) AND (_snuba_tim...
(3 additional frame(s) were not displayed)
...
  File "sentry/snuba/outcomes.py", line 282, in run_outcomes_query
    limit=10000,
  File "sentry/utils/snuba.py", line 654, in raw_query
    [snuba_params], referrer=referrer, use_cache=use_cache, snql_entity=snql_entity
  File "sentry/utils/snuba.py", line 695, in bulk_raw_query
    params, referrer=referrer, use_cache=use_cache, snql_entity=snql_entity
  File "sentry/utils/snuba.py", line 731, in _apply_cache_and_build_results
    query_results = _bulk_snuba_query(map(itemgetter(1), to_query), headers, snql_entity)
  File "sentry/utils/snuba.py", line 808, in _bulk_snuba_query
    error["message"]

Thanks for a little hint :)

No events showing up in 1.13.2 (Sentry 21.4.1)

After getting my DB migrations settled, I found that no new events are showing up in the issue list.

I suspect this change as the root cause: getsentry/self-hosted@4d70217#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3

Furthermore I am not sure if this also needs to be changed:
getsentry/self-hosted@aecc75b#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3

And this:
getsentry/self-hosted@35a45ea#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3

And this:
getsentry/self-hosted@e0dde9c#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3

Please have a look - I am stuck here 🍺

SnubaBootstrap is not run when exchanging the Kafka cluster

I wanted to get rid of my storage settings that I changed manually before #25 was implemented, see also #33 (comment)).
So I renamed the cluster in the template and updated my stack. It took a long time before I got suspicious and looked at the logs.

This pointed me to further investigate the issue:

snuba.utils.streams.backends.abstract.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: event-replacements: Broker: Unknown topic or partition"}

I suspected that the "bootstrap" docker was not executed. I think the reason is this line:

sentry-fargate-cf-stack/cloudformation-template.yaml

Line 5487 in fccdf5c

RerunOnUpdate: "NO"

I started the task manually and the stack finally upgraded with no errors.

The logs show that the missing topics were created:

2021-01-31T22:37:45.586+01:00 | 2021-01-31 21:37:45,585 Topic errors-replacements created
-- | --
  | 2021-01-31T22:37:45.586+01:00 | 2021-01-31 21:37:45,585 Topic event-replacements created

For the next run, I am going to change this line to YES - perhaps this should also be changed in the template, even if it is an edge case.

WDYT?

ServiceSnubaOutcomesConsumer is not restarted after shut down

I noticed that the events stopped processing this morning. Looking at the logs I found

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                          message                                                                           |
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1620013136078 | 2021-05-03 03:38:56,078 Completed processing <Batch: 1 message, open for 1.01 seconds>.                                                                    |
| 1620013138444 | 2021-05-03 03:38:58,444 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013138447 | 2021-05-03 03:38:58,446 Completed processing <Batch: 3 messages, open for 1.03 seconds>.                                                                   |
| 1620013139729 | 2021-05-03 03:38:59,729 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013139729 | 2021-05-03 03:38:59,729 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013139732 | 2021-05-03 03:38:59,732 Completed processing <Batch: 3 messages, open for 1.02 seconds>.                                                                   |
| 1620013141643 | 2021-05-03 03:39:01,643 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013141645 | 2021-05-03 03:39:01,645 Completed processing <Batch: 2 messages, open for 1.01 seconds>.                                                                   |
| 1620013147922 | 2021-05-03 03:39:07,922 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013147922 | 2021-05-03 03:39:07,922 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013147924 | 2021-05-03 03:39:07,924 Completed processing <Batch: 4 messages, open for 1.02 seconds>.                                                                   |
| 1620013149565 | 2021-05-03 03:39:09,565 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013149566 | 2021-05-03 03:39:09,565 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013149568 | 2021-05-03 03:39:09,567 Completed processing <Batch: 4 messages, open for 1.03 seconds>.                                                                   |
| 1620013156473 | 2021-05-03 03:39:16,473 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013156476 | 2021-05-03 03:39:16,475 Completed processing <Batch: 2 messages, open for 1.03 seconds>.                                                                   |
| 1620013157704 | 2021-05-03 03:39:17,703 Error submitting packet, dropping the packet and closing the socket                                                                |
| 1620013157706 | 2021-05-03 03:39:17,706 Completed processing <Batch: 2 messages, open for 1.02 seconds>.                                                                   |
| 1620013160706 | 2021-05-03 03:39:20,706 Caught ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), shutting down... |
| 1620013160717 | Traceback (most recent call last):                                                                                                                         |
| 1620013160717 |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 600, in urlopen                                                            |
| 1620013160718 |     httplib_response = self._make_request(conn, method, url,                                                                                               |
| 1620013160718 |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 387, in _make_request                                                      |
| 1620013160718 |     six.raise_from(e, None)                                                                                                                                |
| 1620013160718 |   File "<string>", line 2, in raise_from                                                                                                                   |
| 1620013160718 |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 383, in _make_request                                                      |
| 1620013160718 |     httplib_response = conn.getresponse()                                                                                                                  |
| 1620013160718 |   File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse                                                |
| 1620013160718 |     rv = real_getresponse(self, *args, **kwargs)                                                                                                           |
| 1620013160718 |   File "/usr/local/lib/python3.8/http/client.py", line 1344, in getresponse                                                                                |
| 1620013160718 |     response.begin()                                                                                                                                       |
| 1620013160718 |   File "/usr/local/lib/python3.8/http/client.py", line 307, in begin                                                                                       |
| 1620013160719 |     version, status, reason = self._read_status()                                                                                                          |
| 1620013160719 |   File "/usr/local/lib/python3.8/http/client.py", line 276, in _read_status                                                                                |
| 1620013160719 |     raise RemoteDisconnected("Remote end closed connection without"                                                                                        |
| 1620013160719 | http.client.RemoteDisconnected: Remote end closed connection without response                                                                              |
| 1620013160719 | During handling of the above exception, another exception occurred:                                                                                        |
| 1620013160719 | Traceback (most recent call last):                                                                                                                         |
| 1620013160719 |   File "/usr/local/bin/snuba", line 33, in <module>                                                                                                        |
| 1620013160719 |     sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())                                                                                      |
| 1620013160719 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__                                                                       |
| 1620013160719 |     return self.main(*args, **kwargs)                                                                                                                      |
| 1620013160719 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main                                                                           |
| 1620013160719 |     rv = self.invoke(ctx)                                                                                                                                  |
| 1620013160719 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke                                                                        |
| 1620013160720 |     return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                |
| 1620013160720 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke                                                                        |
| 1620013160720 |     return ctx.invoke(self.callback, **ctx.params)                                                                                                         |
| 1620013160720 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke                                                                         |
| 1620013160720 |     return callback(*args, **kwargs)                                                                                                                       |
| 1620013160720 |   File "/usr/src/snuba/snuba/cli/consumer.py", line 161, in consumer                                                                                       |
| 1620013160720 |     consumer.run()                                                                                                                                         |
| 1620013160720 |   File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 112, in run                                                                      |
| 1620013160720 |     self._run_once()                                                                                                                                       |
| 1620013160720 |   File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 147, in _run_once                                                                |
| 1620013160720 |     self.__processing_strategy.poll()                                                                                                                      |
| 1620013160720 |   File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/transform.py", line 61, in poll                                                 |
| 1620013160721 |     self.__next_step.poll()                                                                                                                                |
| 1620013160721 |   File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/collect.py", line 122, in poll                                                  |
| 1620013160721 |     self.__close_and_reset_batch()                                                                                                                         |
| 1620013160721 |   File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/collect.py", line 104, in __close_and_reset_batch                               |
| 1620013160721 |     self.__batch.close()                                                                                                                                   |
| 1620013160721 |   File "/usr/src/snuba/snuba/utils/streams/processing/strategies/streaming/collect.py", line 64, in close                                                  |
| 1620013160721 |     self.__step.close()                                                                                                                                    |
| 1620013160721 |   File "/usr/src/snuba/snuba/consumers/consumer.py", line 225, in close                                                                                    |
| 1620013160721 |     self.__insert_batch_writer.close()                                                                                                                     |
| 1620013160721 |   File "/usr/src/snuba/snuba/consumers/consumer.py", line 90, in close                                                                                     |
| 1620013160721 |     self.__writer.write(                                                                                                                                   |
| 1620013160721 |   File "/usr/src/snuba/snuba/clickhouse/http.py", line 245, in write                                                                                       |
| 1620013160721 |     batch.join()                                                                                                                                           |
| 1620013160721 |   File "/usr/src/snuba/snuba/clickhouse/http.py", line 181, in join                                                                                        |
| 1620013160721 |     response = self.__result.result(timeout)                                                                                                               |
| 1620013160721 |   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result                                                                         |
| 1620013160721 |     return self.__get_result()                                                                                                                             |
| 1620013160721 |   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result                                                                   |
| 1620013160722 |     raise self._exception                                                                                                                                  |
| 1620013160722 |   File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run                                                                            |
| 1620013160722 |     result = self.fn(*self.args, **self.kwargs)                                                                                                            |
| 1620013160722 |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 640, in urlopen                                                            |
| 1620013160722 |     retries = retries.increment(method, url, error=e, _pool=self,                                                                                          |
| 1620013160722 |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 368, in increment                                                              |
| 1620013160722 |     raise six.reraise(type(error), error, _stacktrace)                                                                                                     |
| 1620013160722 |   File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 685, in reraise                                                              |
| 1620013160722 |     raise value.with_traceback(tb)                                                                                                                         |
| 1620013160722 |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 600, in urlopen                                                            |
| 1620013160723 |     httplib_response = self._make_request(conn, method, url,                                                                                               |
| 1620013160723 |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 387, in _make_request                                                      |
| 1620013160723 |     six.raise_from(e, None)                                                                                                                                |
| 1620013160723 |   File "<string>", line 2, in raise_from                                                                                                                   |
| 1620013160723 |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 383, in _make_request                                                      |
| 1620013160723 |     httplib_response = conn.getresponse()                                                                                                                  |
| 1620013160723 |   File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse                                                |
| 1620013160723 |     rv = real_getresponse(self, *args, **kwargs)                                                                                                           |
| 1620013160723 |   File "/usr/local/lib/python3.8/http/client.py", line 1344, in getresponse                                                                                |
| 1620013160723 |     response.begin()                                                                                                                                       |
| 1620013160723 |   File "/usr/local/lib/python3.8/http/client.py", line 307, in begin                                                                                       |
| 1620013160723 |     version, status, reason = self._read_status()                                                                                                          |
| 1620013160723 |   File "/usr/local/lib/python3.8/http/client.py", line 276, in _read_status                                                                                |
| 1620013160724 |     raise RemoteDisconnected("Remote end closed connection without"                                                                                        |
| 1620013160724 | urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))                             |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Nevertheless, the status of the task in ECS was still "running" and hence no restarted followed.

I'll check next time this happens by connecting to the respective ECS container and try to look if this can somehow be mitigated, e.g. if the process in the container is really shut down or not.

Feel free to assign this issue to me.

Support for Graviton2 processors

I would like to have support for the new Graviton2 processors, which include the following instance classes:
t4g, c6g, m6g, r6g. Many services like EC2 and RDS are already compatible.

Dropping ECR support broke my stack

Hi,

please re-enable your registry, I am stuck in a loop - I realized something is wrong since "6 days" and tried to update to the new version. Unfortunately I had a typo in one of the image URLs and now I can't rollback due to:

STOPPED (CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref "public.ecr.aws/x7u7j5v7/sentry-relay:1.11.3": public.ecr.aws/x7u7j5v7/sentry-relay:1.11.3: not found)

Of course it worked fine on my stage stack but I messed my prod stack :-/
So it would be perfect if the ECR was back online again.

New deployment of stack does not launch Aurora replicas

Warning for not scaling is:

You have no Aurora Replicas in your DB cluster. Add at least one Aurora Replica for Auto Scaling to start adjusting the number of replicas in your cluster.

Solution -> Create at least 1 Aurora replica to force process to start

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.