Feature: Expanded notification support about awx

[pritho]

Sure, I'm aware of that, but it would be helpful to be able to receive a notification if any of the hosts of a play had a task applied that changed a configuration.
For example if somebody keeps disabling selinux on a certain host.

[mabashian]

From the UI's perspective we're looking at breaking this work down into three sections:

Adding templated content with field substitution
There are still some decisions to be made here. Do we want to give the user the ability to craft specific messages based on the event (Success, Failure, Job Start)? To what extent does the UI need to help with syntax? In some cases, the user may want to overwrite not only the body of the notification but also title (different field lengths). What does that look like in the UI?

• General gist is that each notification type will grow at least one new field that allows the user to overwrite the contents of the notification message. This could be a raw textarea or it could be something "smarter". The expected templating syntax will need to be documented and we may want to include an example or two in the UI. Whatever we decide to use needs to preserve new lines.

Adding http method as well as convenience auth headers to webhook notification type

• Add interface for allowing a user to specify the HTTP method
• Add username/password fields to support generating basic auth headers

Adding support for notify on job start

• This involves adding a third toggle (Start or something similar) to all of our notification lists:

We reuse the same code for all the notification lists throughout the app so we will hopefully only have to modify this in one place.

[matburt]

This also should implement websocket events for notifications (at least notification tests)

[Spredzy]

The three features that made this card are now tested and merged.

• Custom message for notification template
• Basic Auth and custom method for webhook notification
• Allow notification on job_start and not only on job_end

As suggested above I am closing this one card, and for any new feature a new card should be open.

[matburt]

Templated notifications can get a bit tricky because some of our destinations will not allow extended information, but some definitely do and we're not taking advantage of all of them right now (Slack and Hipchat come to mind).

[matburt]

We should come up with a class of notifications that represent the system notifications and make those configurable by system administrators. This should not be made to replace an external monitoring system.

[MichalTaratuta]

I'm surprised that this has not been done yet, Tower is at version 3.2.1 and you can't modify the "template"

[pritho]

Hey, it should also be possible to allow notifications to be sent upon changes, not solely for success or failed. This would allow for security monitoring and reporting of infrastructure/configuration changes.

[wenottingham]

Playbooks only have a success and failure state; 'changed' only applies to hosts.

[eviln1]

Hi,
It would be awesome if there was a support for Grafana Annotations (http://docs.grafana.org/http_api/annotations/)

The easiest form would be basically a webhook with more features: allow selection the http method, and a field where I could write the POST payload as a Jinja template, and substitute some variables.

[MBcom]

Is it possible to get the output of the debug module/ stdout into the e-mail notfications?
We have a playbook which shows us hosts who need updates and it would be great if we could get an email containing the complete ansible output for all hosts. The mail module sends us a mail for every host and this is not the behaviour we want. And the debug module output is not shown in the notification mail.

[wenottingham]

Is it possible to get the output of the debug module/ stdout into the e-mail notfications?

This is highly unlikely to ever happen, just due to the scaling problem it causes if you're trying to schlep a 10MB log file around.

[MBcom]

@wenottingham ok your are right generally it is not a good idea but it would be nice if there were checkbox to enable this feature anyhow - because not every environment contains thousands of servers
Or could you tell me the code position where the notifactions are generated - so I could change it on my own for our company?

[matburt]

You can find the notification backends here:

https://github.com/ansible/awx/tree/devel/awx/main/notifications

If you have any other questions related to this specific topic it'd be best to take it up on the mailing list or IRC.

[MBcom]

@matburt thank you for your fast reply
Can you tell me where the json body is generated? I am new in python and I could not find the concrete code position...

I changed in /usr/lib/python2.7/site-packages/awx/main/models/jobs (in docker containers) (

awx/awx/main/models/jobs.py

Lines 685 to 692 in 07680dd

	for h in self.job_host_summaries.all():
	all_hosts[h.host_name] = dict(failed=h.failed,
	changed=h.changed,
	dark=h.dark,
	failures=h.failures,
	ok=h.ok,
	processed=h.processed,
	skipped=h.skipped)

) to

all_hosts[h.host_name] = dict(failed=h.failed,
                                          changed=h.changed,
                                          dark=h.dark,
                                          failures=h.failures,
                                          ok=h.ok,
                                          processed=h.processed,
                                          skipped=h.skipped,
                                          stdout=h.stdout)

but this does not put the stdout in the notification.

(building from source does not works for me, the playbook exits with an error: unexpected parameter: 'stream' in /usr/lib/python2.7/dist-packages/ansible/modules/cloud/docker/docker_image.py Line 602)

[ryanpetrello]

@MBcom stdout for the job is not stored in a host-specific manner in the way you're attempting to access it; if you wanted to include the stdout for the entire job, you'd need to add it here:

awx/awx/main/models/jobs.py

Line 699 in 07680dd

hosts=all_hosts))

e.g.,

...
hosts=all_hosts,
stdout=self.result_stdout_raw_handle().read().decode('utf-8'))

Keep in mind that these interfaces are subject to change (future upgrades of awx likely will break with your one-off changes), and as @wenottingham mentioned above, this is likely not the type of change we'd introduce to awx due to the performance implications.

[MBcom]

An working AWX with mail notifications containing the ansible output can be found at https://github.com/mbcom/awx/

@ryanpetrello and @matburt - thanks for helping me on this issue

+1 Must needed feature

[ryanpetrello]

FYI, we've just released a new version of awx (https://github.com/ansible/awx/releases/tag/3.0.0) which includes support for notifications using the Grafana annotations API.

[lmfreitas]

Hi,
I was talking about this notifications and the graph in the dashboard for job status

Could it be added a Changed Notification?

I use a playbook to update servers YUM plugin returns Changed, Failed and Success, most of notifications are Success so nothing has be looked upon, i have to click every single day in the job so see if its has happen any change.

[wenottingham]

Ah. For that, the playbook itself doesn't return an overall 'changed' status.... you'd almost need a per-host notification.

[perzizzle]

I would be interested in being able to control the http method, http body via a template, and authentication methods. Support submitted #3660 for me.

[ryanpetrello]

Implementation Notes for "On Start" Notification Templates

Discussed implementation details of this feature w/ @beeankha today. Here are some highlights of the work to do:

1. Add a new relationship to awx.main.models.base.NotificationFieldsModel. This field is inherited by UnifiedJobTemplate and Organization.

2. Create a new migration (using awx-manage makemigrations so that the new many-to-many relationship tables are created for housing this data.

3. Add new API endpoints to support these new associate/disassociate relationships:

   • /api/v2/job_templates/N/notification_templates_start
   • /api/v2/projects/N/notification_templates_start
   • /api/v2/inventory_sources/N/notification_templates_start
   • /api/v2/organizations/N/notification_templates_start

4. Write API tests to verify the above endpoints properly affect the database (sending associate/disassociate requests actually affect the database in the anticipated manner).

At this point in time, the UI probably has what they need to start integrating with the API. @mabashian how do you feel about this?

5. In terms of actually implementing the notification under the hood, the plan is to augment the awx.main.models.notification.JobNotificationMixin().send_notification_templates() function so that it can also handle the "running" state.

[ryanpetrello]

Implementation Notes for Custom HTTP Method + Basic Auth Support

Discussed implementation details of this feature w/ @beeankha today. Here are some highlights of the work to do:

Custom HTTP Method

1. Add a new init_parameter to awx.main.notifications.webhook_backend.WebhookBackend. This field will capture a new "HTTP Method" field that will be configurable from the UI. Ideally, this field will not be free-form text, but instead will be limited to a set of choices (likely to PUT or POST). We may need to add some form of support to the init_parameter for specifying parameters, and we'll likely need code somewhere (potentially in the serializer for templates?) that ensures users don't pass data that isn't PUT or POST.
2. Users should be able to edit this method field for notification templates, and it should affect the HTTP request method utilized when the notification is sent. cc @mabashian @trahman73 the UI may need smarts to render this metadata as dropdown/selectable instead of a plain text field when adding/editing a Notification Template.
3. When upgrading AWX, any web hook notification templates created before this change will need to be verified to make sure they still fall back to using POST by default (the prior behavior).

Basic Auth Support

1. Add two new init_parameters to awx.main.notifications.webhook_backend.WebhookBackend, one for the username, and one for the password.
2. Update the code that sends HTTP requests so that it properly passes the username into password into the requests library so that the proper Authorization: Basic ... header is generated for the request.
3. Under the hood, any notification template that's saved in the database should store the password in an encrypted (non plaintext) fashion. note: this should already be implemented via eb3d663, we just need to confirm it works. We should make a web hook notification template w/ a password and verify that:
   a. the underlying data in the main_notificationtemplate table is encrypted.
   b. the API doesn't return decrypted data at /api/v2/notification_templates/N/
   c. the unencrypted value is actually what's sent to the web hook destination when the notification ends up being sent

@mabashian does the UI need special code to handle these new parameters, or is it smart enough to just draw the fields based on the fields defined on the template?

[jladdjr]

Added checklist to description so we can mark off progress on the different work items associated with this ticket. Marked 'templated notifications' and 'notifications on job start' as complete. Not sure about the status of the other items.

[jladdjr]

I removed the in_progress label and added needs_review. To my knowledge, nothing is being actively worked on for this ticket. (If I'm wrong about that, please accept my apologies .. and also leave an update 😄)

[ryanpetrello]

@jladdjr @beeankha have we finished everything we agreed on re: notifications in the next release?

If so, @wenottingham maybe we should put this in state:needs_test and track any other work we didn't do as additional issues?

[jladdjr]

@ryanpetrello - I'm still working with @tvo318 on docs, otherwise that's a yes for me

[ryanpetrello]

@wenottingham can you file new distinct issues for these if you still care about tracking them?

[wenottingham]

If they end up being needed, I can; this can be closed when it's done.

[fxfitz]

For what it's worth, we're also interested on an on_changed notification for a playbook summary. Just like someone suggested above, we really want to know when our iptables or our selinux playbooks change on any one of our hosts.

While the playbook doesn't have a "changed", you should be able to see if the playbook changed anything by filtering through the hosts/tasks the changed, no?

[matburt]

For what it's worth, we're also interested on an on_changed notification for a playbook summary.

You can probably get this by just looking at the summary event. I'm not sure if any of this is super relevant here. You can take this event as a webhook and then introspect the summary event to know if the host you were talking to changed in any way.