aws-samples / asynchronous-messaging-workshop

The Asynchronous Messaging Workshop is a collection of individual labs covering different aspects and patterns of asynchronous messaging.

Asynchronous Messaging Workshop

Overview

The Asynchronous Messaging Workshop is a collection of individual labs covering different aspects and patterns of asynchronous messaging. You will use Amazon SQS, Amazon SNS and AWS Step Functions during the workshop extensively and we will use AWS Lambda, AWS SAM and the AWS Management Console as well. Each lab can be done separately and there is no dependency between the rounds. Each lab takes between 15 - 30 minutes, depending which lab you are executing and whether you will use the AWS Console or AWS SAM.

• Level: 300

• Duration: Each lab takes 15 - 30 minutes:

  • Lab 0: Set-up the workshop environment
  • Lab 1: Fan-Out & Message Filtering
  • Lab 2: Topic-Queue Chaining & Load Balancing
  • Lab 3: Scatter-Gather Pattern
  • Lab 4: Orchestration and Coordination

• Prerequisites:

  • AWS Account
  • AWS IAM User (with admin permissions)

Get started

To get started, click here to proceed to the workshop site.

asynchronous-messaging-workshop's Issues

Please simplify instructions for getting SNS topic ARN

in https://async-messaging.workshop.aws/fan-out-and-message-filtering/update-unicorn-management-service/update-unicorn-management-service.html it would be clearer to simply ask the user to copy-paste the whole topic ARN from SNS.
The current instructions encourage the user to construct the ARN one piece at a time for no reason.

and similarly confusing is the lambda function code section, which tells the user to add code (and goes to immense trouble to hide that code in cheat sheets) when the code is already present in the lambda (just commented).

the instructions also neglect to tell the user to click Deploy after the function changes are made

ecs container error(maybe)

in LAB-1-9.Test fan-out&messaging filter, I found out that ecs task came across error like the below.
can you check out reason for this when you get a chance?

2020-03-19T14:02:09.868+09:00 | INFO:web:
-- | --
  | 2020-03-19T14:02:09.868+09:00 | processing message: {"from": "Berlin", "to": "Frankfurt", "duration": 420, "distance": 600, "customer": "cmr", "fare": 256.5, "id": "c1e00329-188d-4f81-b741-dc7e367a077e"} ...
  | 2020-03-19T14:02:09.868+09:00 | ERROR:web:Exception on / [POST]
  | 2020-03-19T14:02:09.868+09:00 | Traceback (most recent call last):
  | 2020-03-19T14:02:09.868+09:00 | File "/usr/lib/python2.7/site-packages/flask/app.py", line 2446, in wsgi_app
  | 2020-03-19T14:02:09.868+09:00 | response = self.full_dispatch_request()
  | 2020-03-19T14:02:09.868+09:00 | File "/usr/lib/python2.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
  | 2020-03-19T14:02:09.868+09:00 | rv = self.handle_user_exception(e)
  | 2020-03-19T14:02:09.868+09:00 | File "/usr/lib/python2.7/site-packages/flask/app.py", line 1820, in handle_user_exception
  | 2020-03-19T14:02:09.868+09:00 | reraise(exc_type, exc_value, tb)
  | 2020-03-19T14:02:09.868+09:00 | File "/usr/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
  | 2020-03-19T14:02:09.868+09:00 | rv = self.dispatch_request()
  | 2020-03-19T14:02:09.868+09:00 | File "/usr/lib/python2.7/site-packages/flask/app.py", line 1935, in dispatch_request
  | 2020-03-19T14:02:09.868+09:00 | return self.view_functions[rule.endpoint](**req.view_args)
  | 2020-03-19T14:02:09.868+09:00 | File "/app/web/__init__.py", line 45, in index
  | 2020-03-19T14:02:09.868+09:00 | message = json.loads(data)['Message']
  | 2020-03-19T14:02:09.868+09:00 | KeyError: 'Message'
  | 2020-03-19T14:02:09.868+09:00 | INFO:werkzeug:10.11.0.170 - - [19/Mar/2020 05:02:09] "POST / HTTP/1.1" 500 -
  | 2020-03-19T14:02:27.246+09:00 | INFO:werkzeug:10.11.1.82 - - [19/Mar/2020 05:02:27] "GET / HTTP/1.1" 200 -

Adjust the "create SNS topic" step - Lab 1

After the release of the new feature "SNS FIFO Topics", the default topic type became FIFO when users are creating one by AWS Console. The problem occurs in Labs 1, when workshop attendees create a new topic using the default values, in the next step it is not possible to create the subscription using HTTP protocol.

The image should be updated as follows:

issues with setting up of Cloud9

I followed instructions on: https://async-messaging.workshop.aws/prerequisites/prerequisites-2/prerequisites-2.html

I tried to run:

cd ~/environment/wild-rydes-async-messaging/code/lab-0
chmod +x configureCloud9.sh
./configureCloud9.sh

however I ran into 2 issues:

1. the folder did not exist
2. after sorting that out, the folder name was incorrect

the fix:
Inside Cloud9's terminal, manually run git clone https://github.com/aws-samples/asynchronous-messaging-workshop
then, cd into that directory (instead of wild-rydes-async-messaging per the instructions on the page)
then, proceed with the chmod and running of configureCloud9.sh (some lines in the script depend on the old name)

so, then, rename the folder back from asynchronous-messaging-workshop to wild-rydes-async-messaging

Scatter-Gather: Is the endpoint in the architecture diagram the intended one?

In /scatter-gather.html, the endpoint submit-ride-completion is written right to the mobile icon. Shouldn't a reference to RequestForQuotesService and QueryForQuotesService be written here?

Prerequisites template still installing Amazon Linux 1 Cloud9 instance.

Should be installing AL2 C9 instance

Lab-2 message retry information incorrect. Addition of DLQ steps might also be beneficial

In the following lab-2 guide:
https://async-messaging.workshop.aws/topic-queue-chaining-and-load-balancer/test-topic-queue-chaining-and-load-balancer/test-topic-queue-chaining-and-load-balancing.html#3-validate-the-message-reception

It reports:

You should also see some log entries, indicating a failed message processing. Shortly after, you should see the message redelivery from Amazon SNS and the successful message processing log entry.

In lab-2 we have configured SQS queues as subscribers to the SNS topics and so it isn't SNS re-sending the message, but the Lambda consumer keep retrieving and re-trying the message from the queue. It would be good to update the docs to avoid confusion.

Also - it might be beneficial to add steps for enabling a DLQ for one of the SQS queues, after we have updated the logic of one of the backend lambdas to always fail (to simulate a persistent message failure and how to handle them).

3 small bugs in Lab2

• Bug with raw message format delivery:
  When retesting please make sure both “SAM-way” and “Console-way” is tested. We have found a bug in the workshop last week preventing successful completion of “Console-way”. This was about “raw message delivery” for SNS subscriptions which was enabled in “SAM-way” but disabled in “Console-way”.
  Also the way “Console-way” suggests creating the subscription does not allow for setting “raw message delivery” easily. More instruction is required.
  On the other hand disabling “raw message delivery” breaks lambda functions processing messages.
  I thought fixing the lambdas would be easier than adding more instructions so I patched lambdas in my fork in quick & dirty way.

• Clean-up instruction for “Console-way” is incomplete and will result in CloudFormation stack deletion failure. I believe similar issues has been fixed in lab-1 – not sure why those changes are not applied consistently to all labs?

• Minor documentation problems indicating IAM role should have 3 policies while it should actually be 2 policies. Not a biggie but does confuse the attendees.

Lab clean-up instruction for “Console-way” are incomplete

Clean-up instruction for “Console-way” is incomplete and will result in CloudFormation stack deletion failure. I believe similar issues has been fixed in lab-1 – not sure why those changes are not applied consistently to all labs?

Feature request: Upgrade C9 to use AL2

Use Amazon Linux 2 as the Cloud9 image.

Installing cfn-lint fails when running cinfigureCloud9.sh

Installing cfn-lint fails with

ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.`

when running the configureCloud9.sh

sudo pip install cfn-lint
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Collecting cfn-lint
  Downloading cfn-lint-0.51.0.tar.gz (6.2 MB)
     |████████████████████████████████| 6.2 MB 6.2 MB/s 
Requirement already satisfied: six>=1.11 in /usr/lib/python2.7/site-packages (from cfn-lint) (1.16.0)
Collecting aws-sam-translator>=1.36.0
  Downloading aws_sam_translator-1.36.0-py2-none-any.whl (222 kB)
     |████████████████████████████████| 222 kB 24.7 MB/s 
Collecting jsonschema~=3.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 7.6 MB/s 
Collecting junit-xml~=1.9
  Downloading junit_xml-1.9-py2.py3-none-any.whl (7.1 kB)
Collecting importlib_resources<4,>=1.4
  Downloading importlib_resources-3.3.1-py2.py3-none-any.whl (26 kB)
Collecting pyyaml>=5.4
  Downloading PyYAML-5.4.1-cp27-cp27mu-manylinux1_x86_64.whl (574 kB)
     |████████████████████████████████| 574 kB 18.3 MB/s 
Collecting networkx<=2.2
  Downloading networkx-2.2.zip (1.7 MB)
     |████████████████████████████████| 1.7 MB 28.3 MB/s 
Collecting pyrsistent<=0.16.0
  Downloading pyrsistent-0.16.0.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 34.1 MB/s 
Collecting pathlib2>=2.3.0
  Downloading pathlib2-2.3.5-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: jsonpatch in /usr/lib/python2.7/site-packages (from cfn-lint) (1.2)
Collecting boto3~=1.5
  Downloading boto3-1.17.97-py2.py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 23.1 MB/s 
Requirement already satisfied: enum34~=1.1; python_version < "3.4" in /usr/lib/python2.7/site-packages (from aws-sam-translator>=1.36.0->cfn-lint) (1.1.10)
Collecting functools32; python_version < "3"
  Downloading functools32-3.2.3-2.tar.gz (31 kB)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from jsonschema~=3.0->cfn-lint) (41.2.0)
Collecting importlib-metadata; python_version < "3.8"
  Downloading importlib_metadata-2.1.1-py2.py3-none-any.whl (10 kB)
Collecting attrs>=17.4.0
  Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 4.0 MB/s 
Collecting zipp>=0.4; python_version < "3.8"
  Downloading zipp-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting contextlib2; python_version < "3"
  Downloading contextlib2-0.6.0.post1-py2.py3-none-any.whl (9.8 kB)
Requirement already satisfied: singledispatch; python_version < "3.4" in /usr/lib/python2.7/site-packages (from importlib_resources<4,>=1.4->cfn-lint) (3.6.2)
Collecting typing; python_version < "3.5"
  Downloading typing-3.10.0.0-py2-none-any.whl (26 kB)
Collecting decorator>=4.3.0
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting scandir; python_version < "3.5"
  Downloading scandir-1.10.0.tar.gz (33 kB)
Requirement already satisfied: jsonpointer>=1.0 in /usr/lib/python2.7/site-packages (from jsonpatch->cfn-lint) (1.9)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/lib/python2.7/site-packages (from boto3~=1.5->aws-sam-translator>=1.36.0->cfn-lint) (0.9.3)
Collecting botocore<1.21.0,>=1.20.97
  Downloading botocore-1.20.97-py2.py3-none-any.whl (7.6 MB)
     |████████████████████████████████| 7.6 MB 19.7 MB/s 
Requirement already satisfied: s3transfer<0.5.0,>=0.4.0 in /usr/lib/python2.7/site-packages (from boto3~=1.5->aws-sam-translator>=1.36.0->cfn-lint) (0.4.2)
Requirement already satisfied: configparser>=3.5; python_version < "3" in /usr/lib/python2.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema~=3.0->cfn-lint) (4.0.2)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python2.7/site-packages (from botocore<1.21.0,>=1.20.97->boto3~=1.5->aws-sam-translator>=1.36.0->cfn-lint) (2.6.0)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python2.7/site-packages (from botocore<1.21.0,>=1.20.97->boto3~=1.5->aws-sam-translator>=1.36.0->cfn-lint) (1.25.7)
Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.7" in /usr/lib/python2.7/site-packages (from s3transfer<0.5.0,>=0.4.0->boto3~=1.5->aws-sam-translator>=1.36.0->cfn-lint) (3.0.5)
Using legacy 'setup.py install' for cfn-lint, since package 'wheel' is not installed.
Using legacy 'setup.py install' for networkx, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pyrsistent, since package 'wheel' is not installed.
Using legacy 'setup.py install' for functools32, since package 'wheel' is not installed.
Using legacy 'setup.py install' for scandir, since package 'wheel' is not installed.
Installing collected packages: botocore, boto3, functools32, pyrsistent, contextlib2, scandir, pathlib2, zipp, importlib-metadata, attrs, jsonschema, aws-sam-translator, junit-xml, typing, importlib-resources, pyyaml, decorator, networkx, cfn-lint
  Attempting uninstall: botocore
    Found existing installation: botocore 1.20.94
    Uninstalling botocore-1.20.94:
      Successfully uninstalled botocore-1.20.94
    Running setup.py install for functools32 ... done
    Running setup.py install for pyrsistent ... done
    Running setup.py install for scandir ... done
  Attempting uninstall: jsonschema
    Found existing installation: jsonschema 2.5.1
    Uninstalling jsonschema-2.5.1:
      Successfully uninstalled jsonschema-2.5.1
  Attempting uninstall: pyyaml
    Found existing installation: PyYAML 3.10
ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Add instructions to clean up objects created during the lab (especially IAM roles that prevent CFN delete)

Lab 1 cleanup fails due to the custom IAM role:
Cannot delete entity, must delete policies first. (Service: AmazonIdentityManagement; Status Code: 409; Error Code: DeleteConflict; ...
The following resource(s) failed to delete: [SubmitRideCompletionFunctionRole].
so I think this role needs to be deleted by hand.

Similarly, all manually created artifacts need delete instrucitons.
For example: the SNS topic created in lab 1 stays untouched, so following Lab 2 instructions brings me back to SNS with the topic and subscriptions I'm supposed to create already there...

Update needed to SNS creation step screenshot/instrucitons

It looks like the SNS creation step (https://async-messaging.workshop.aws/fan-out-and-message-filtering/create-sns-topic/create-sns-topic.html) has fallen out of sync with the AWS Console (which now looks like this and requires you to select FIFO or Standard).
Could you please update the workshop to reflect this (and add the extra step of switching to Standard or adding the .fifo suffix to the queue name?

Add more data target screenshots in lab 1

it might be nice to point the user to a few more places the use can see their data (like Dynamo)

Resources: Replace link to deck with more generic version

re-sync SAM instructions to new CLI prompts

I've noticed the SAM default instructions were slightly wrong (and somewhat confusing).
The instructions say "Confirm the first 4 proposed arguments by hitting ENTER. When you get asked SubmitRideCompletionFunction may not have authorization defined, Is this okay? [y/N]:, enter y and hit ENTER again 2 times."
but there are now 3, not 2 prompts at the end.

Cloud9 is missing Python 3.7 (and Linux2)

sam build now throws this error for all labs:
Error: PythonPipBuilder:Validation - Binary validation failed for python, searched for python in following locations : ['/usr/bin/python'] which did not satisfy constraints for runtime: python3.7. Do you have python for runtime: python3.7 on your PATH?

I think you need to specify Amazon Linux 2 per https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloud9-environmentec2.html in the C9 template.

Lab 2: Instruction issues

Step 5 in https://async-messaging.workshop.aws/topic-queue-chaining-and-load-balancer/create-customer-accounting-service-subscription/create-customer-accounting-service-subscription.html contains

Step 6 in https://async-messaging.workshop.aws/topic-queue-chaining-and-load-balancer/create-extraordinary-rides-service-subscription/create-extraordinary-rides-service-subscription.html have wrong titles referring to Notification Service while it should be Accounting Service and Extraordinary Rides Service respectively.