alerta / alerta

Alerta monitoring system

Alerta Release 9.1

The Alerta monitoring tool was developed with the following aims in mind:

• distributed and de-coupled so that it is SCALABLE
• minimal CONFIGURATION that easily accepts alerts from any source
• quick at-a-glance VISUALISATION with drill-down to detail

---

Requirements

Release 9 only supports Python 3.9 or higher.

The only mandatory dependency is MongoDB or PostgreSQL. Everything else is optional.

• Postgres version 13 or better
• MongoDB version 6.0 or better

Installation

To install MongoDB on Debian/Ubuntu run:

$ sudo apt-get install -y mongodb-org
$ mongod

To install MongoDB on CentOS/RHEL run:

$ sudo yum install -y mongodb
$ mongod

To install the Alerta server and client run:

$ pip install alerta-server alerta
$ alertad run

To install the web console run:

$ wget https://github.com/alerta/alerta-webui/releases/latest/download/alerta-webui.tar.gz
$ tar zxvf alerta-webui.tar.gz
$ cd dist
$ python3 -m http.server 8000

>> browse to http://localhost:8000

Docker

Alerta and MongoDB can also run using Docker containers, see alerta/docker-alerta.

Configuration

To configure the alertad server override the default settings in /etc/alertad.conf or using ALERTA_SVR_CONF_FILE environment variable::

$ ALERTA_SVR_CONF_FILE=~/.alertad.conf
$ echo "DEBUG=True" > $ALERTA_SVR_CONF_FILE

Documentation

More information on configuration and other aspects of alerta can be found at http://docs.alerta.io

Development

To run in development mode, listening on port 5000:

$ export FLASK_APP=alerta FLASK_DEBUG=1
$ pip install -e .
$ flask run

To run in development mode, listening on port 8080, using Postgres and reporting errors to Sentry:

$ export FLASK_APP=alerta FLASK_DEBUG=1
$ export DATABASE_URL=postgres://localhost:5432/alerta5
$ export SENTRY_DSN=https://8b56098250544fb78b9578d8af2a7e13:[email protected]/153768
$ pip install -e .[postgres]
$ flask run --debugger --port 8080 --with-threads --reload

Troubleshooting

Enable debug log output by setting DEBUG=True in the API server configuration:

DEBUG=True

LOG_HANDLERS = ['console','file']
LOG_FORMAT = 'verbose'
LOG_FILE = '$HOME/alertad.log'

It can also be helpful to check the web browser developer console for JavaScript logging, network problems and API error responses.

Tests

To run the all the tests there must be a local Postgres and MongoDB database running. Then run:

$ TOXENV=ALL make test

To just run the Postgres or MongoDB tests run:

$ TOXENV=postgres make test
$ TOXENV=mongodb make test

To run a single test run something like:

$ TOXENV="mongodb -- tests/test_search.py::QueryParserTestCase::test_boolean_operators" make test
$ TOXENV="postgres -- tests/test_queryparser.py::PostgresQueryTestCase::test_boolean_operators" make test

Cloud Deployment

Alerta can be deployed to the cloud easily using Heroku https://github.com/alerta/heroku-api-alerta, AWS EC2 https://github.com/alerta/alerta-cloudformation, or Google Cloud Platform https://github.com/alerta/gcloud-api-alerta

License

Alerta monitoring system and console
Copyright 2012-2023 Nick Satterly

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

alerta's Issues

Email Notifications?

Hello Chaps,

Do you have plans for or have/seen contributions for sending email/html notifications?

T

Alertad crashes when you go to "About" in alerta web

This is what alertad is showing when you go to About in alerta web.

5.172.237.230 - - [13/Jun/2015 19:50:25] "GET /management/status HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/local/lib/python2.7/dist-packages/flask_cors/extension.py", line 110, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask_cors/extension.py", line 110, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/flask_cors/decorator.py", line 120, in wrapped_function
    resp = make_response(f(*args, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/alerta_server-4.4.4-py2.7.egg/alerta/app/management/views.py", line 121, in status
    total_alert_gauge.set(db.get_count())
  File "/usr/local/lib/python2.7/dist-packages/alerta_server-4.4.4-py2.7.egg/alerta/app/metrics.py", line 38, in set
    True
  File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 635, in update_one
    common.validate_ok_for_update(update)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/common.py", line 377, in validate_ok_for_update
    raise ValueError('update only works with $ operators')
ValueError: update only works with $ operators

can't close an alert

Hi, maybe I am missing something but I can't find out how to close an alert from the UI.

Is a feature that is not just implemented yet?

No big deal if that's the case, I've forked and I can try to get it done.

Alerts not showing in web console

I have installed alerta and web console and able to send alerts to it. Also able to list the alerts from CLI using 'alerta query'. Anyhow I am unable to see these alerts from the web console

So I created a conf file with the following content -

[DEFAULT]
timezone = Australia/Sydney
output = json

[profile production]
endpoint = https://api.alerta.io
key = demo-key

[profile development]
endpoint = http://localhost:8080
debug = yes

And when starting the daemon alertad with the above conf, I am getting the following error -

Traceback (most recent call last):
File "/usr/local/bin/alertad", line 9, in
load_entry_point('alerta-server==4.5.1', 'console_scripts', 'alertad')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 356, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2476, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2190, in load
['name'])
File "/usr/local/lib/python2.7/dist-packages/alerta/app/init.py", line 15, in
app.config.from_pyfile('/etc/alertad.conf', silent=True)
File "/usr/local/lib/python2.7/dist-packages/flask/config.py", line 129, in from_pyfile
exec(compile(config_file.read(), filename, 'exec'), d.dict)
File "/etc/alertad.conf", line 5
[ profile Production ]
^

Running alerta query top-like?

Is there a way to run alerta query like the Unix command top with auto updates in the terminal?

API key fail to generate

When trying to generate a new api key, I get the following error in alertad :

[Tue Apr 21 19:02:19 2015] [error] 2015-04-21 19:02:19,883 - alerta.app.cors[24193]: INFO - Request from Origin:http://alerta.emv2.com, setting Access-Control-Allow-Origin:http://alerta.emv2.com
[Tue Apr 21 19:02:19 2015] [error] 2015-04-21 19:02:19,890 - alerta.app[24193]: ERROR - Exception on /key [POST]
[Tue Apr 21 19:02:19 2015] [error] Traceback (most recent call last):
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/flask/app.py", line 1817, in wsgi_app
[Tue Apr 21 19:02:19 2015] [error]     response = self.full_dispatch_request()
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/flask/app.py", line 1477, in full_dispatch_request
[Tue Apr 21 19:02:19 2015] [error]     rv = self.handle_user_exception(e)
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/flask_cors/extension.py", line 110, in wrapped_function
[Tue Apr 21 19:02:19 2015] [error]     return cors_after_request(app.make_response(f(*args, **kwargs)))
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/flask/app.py", line 1381, in handle_user_exception
[Tue Apr 21 19:02:19 2015] [error]     reraise(exc_type, exc_value, tb)
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/flask/app.py", line 1475, in full_dispatch_request
[Tue Apr 21 19:02:19 2015] [error]     rv = self.dispatch_request()
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/flask/app.py", line 1461, in dispatch_request
[Tue Apr 21 19:02:19 2015] [error]     return self.view_functions[rule.endpoint](**req.view_args)
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/flask_cors/decorator.py", line 120, in wrapped_function
[Tue Apr 21 19:02:19 2015] [error]     resp = make_response(f(*args, **kwargs))
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/alerta/app/auth.py", line 94, in decorated
[Tue Apr 21 19:02:19 2015] [error]     payload = parse_token(token)
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/alerta/app/auth.py", line 53, in parse_token
[Tue Apr 21 19:02:19 2015] [error]     return jwt.decode(token, key=app.config['SECRET_KEY'], audience=app.config['OAUTH2_CLIENT_ID'])
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/jwt/api.py", line 137, in decode
[Tue Apr 21 19:02:19 2015] [error]     self._validate_claims(payload, options=merged_options, **kwargs)
[Tue Apr 21 19:02:19 2015] [error]   File "/opt/alerta/lib/python2.6/site-packages/jwt/api.py", line 216, in _validate_claims
[Tue Apr 21 19:02:19 2015] [error]     raise InvalidIssuedAtError('Issued At claim (iat) cannot be in the future.')
[Tue Apr 21 19:02:19 2015] [error] InvalidIssuedAtError: Issued At claim (iat) cannot be in the future.
[Tue Apr 21 19:02:19 2015] [error] 2015-04-21 19:02:19,891 - alerta.app.cors[24193]: INFO - Request from Origin:http://alerta.emv2.com, setting Access-Control-Allow-Origin:http://alerta.emv2.com

allow setting of mongodb user/password via environment variables

I don't see an easy way to use an environment variable to set the mongodb password.

thresholdInfo should be updated on change of severity

The thresholdInfo attribute remains unchanged when the severity changes and this could create confusion when it is included in emails reporting the new severity.

debian packaging (setup.py - contrib / debian / README.md)

Hi

Not sure if this is documented elsewhere, the package 'python-stdeb' is required for:

python setup.py --command-packages stdeb.command sdist_dsc

to succeed.

Automatically create indexes when Pymongo 3 is fixed

Bug report https://jira.mongodb.org/browse/PYTHON-961

Integration with Opsweekly

can we integrate with Opsweekly https://github.com/etsy/opsweekly for alert reporting

Links in API landing page broken for reverse proxied installations

PyMongo 3.0 admin calls failing when run using Apache WSGI

[Sat Jun 20 12:53:00.260834 2015] [:error] [pid 19124:tid 140441230903040] 2015-06-20 12:53:00,260 - alerta.app[19124]: INFO - Starting alerta version 4.4.7 ...
[Sat Jun 20 12:53:00.322813 2015] [:error] [pid 19124:tid 140441230903040] 2015-06-20 12:53:00,322 - alerta.app[19124]: DEBUG - Connected to mongodb://localhost:27017/monitoring
[Sat Jun 20 12:53:30.368402 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204] mod_wsgi (pid=19124): Target WSGI script '/var/www/api.wsgi' cannot be loaded as Python module.
[Sat Jun 20 12:53:30.369482 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204] mod_wsgi (pid=19124): Exception occurred processing WSGI script '/var/www/api.wsgi'.
[Sat Jun 20 12:53:30.369865 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204] Traceback (most recent call last):
[Sat Jun 20 12:53:30.370266 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/var/www/api.wsgi", line 4, in <module>
[Sat Jun 20 12:53:30.370691 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     from alerta.app import app as application
[Sat Jun 20 12:53:30.370976 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/opt/alerta/lib/python2.7/site-packages/alerta/app/__init__.py", line 49, in <module>
[Sat Jun 20 12:53:30.371371 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     db = Mongo()
[Sat Jun 20 12:53:30.371839 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/opt/alerta/lib/python2.7/site-packages/alerta/app/database.py", line 81, in __init__
[Sat Jun 20 12:53:30.372459 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     self.version = self.db.collection_names()
[Sat Jun 20 12:53:30.372813 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/opt/alerta/lib/python2.7/site-packages/pymongo/database.py", line 488, in collection_names
[Sat Jun 20 12:53:30.373313 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     ReadPreference.PRIMARY) as (sock_info, slave_okay):
[Sat Jun 20 12:53:30.373520 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
[Sat Jun 20 12:53:30.373826 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     return self.gen.next()
[Sat Jun 20 12:53:30.374027 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/opt/alerta/lib/python2.7/site-packages/pymongo/mongo_client.py", line 699, in _socket_for_reads
[Sat Jun 20 12:53:30.374501 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     with self._get_socket(read_preference) as sock_info:
[Sat Jun 20 12:53:30.374706 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
[Sat Jun 20 12:53:30.374978 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     return self.gen.next()
[Sat Jun 20 12:53:30.375184 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/opt/alerta/lib/python2.7/site-packages/pymongo/mongo_client.py", line 663, in _get_socket
[Sat Jun 20 12:53:30.375440 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     server = self._get_topology().select_server(selector)
[Sat Jun 20 12:53:30.375661 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/opt/alerta/lib/python2.7/site-packages/pymongo/topology.py", line 121, in select_server
[Sat Jun 20 12:53:30.376016 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     address))
[Sat Jun 20 12:53:30.376039 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]   File "/opt/alerta/lib/python2.7/site-packages/pymongo/topology.py", line 97, in select_servers
[Sat Jun 20 12:53:30.376062 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204]     self._error_message(selector))
[Sat Jun 20 12:53:30.376095 2015] [:error] [pid 19124:tid 140441230903040] [remote 192.168.0.1:8204] ServerSelectionTimeoutError: No servers found yet

PyMongo 3.0 causing alerta to crash

2015-06-10 13:23:30,805 - alerta.app[14679]: ERROR - Failed to connect to AMQP transport mongodb://localhost:27017/kombu: Unknown option auto_start_request
2015-06-10 13:23:30,805 - alerta.app[14679]: ERROR - Server plug-in 'amqp' could not be loaded:

I am using mongo 2.4.9.

Then when I do alerta query, I get:

2015-06-10 13:24:08,244 - alerta.shell - ERROR - 500 Server Error: INTERNAL SERVER ERROR

Mixed content error while accessing alerta

I am getting a mixed content error - Blocked loading mixed active content "http://alerta.mydomain.com:8080/alerts/count?status=open"

http://alerta.mydomain.com is already redirected (301) to https://alerta.mydomain.com and working

The error is same with default config (ie. removed /etc/alerta.conf) and with alerta.conf. Also tried with both environments development and production

Anyhow my config is as follows:-

[DEFAULT]
timezone = India
output = json

[profile production]
endpoint = https://alerta.mydomain.com
key = demo-key

[profile development]
endpoint = https://alerta.mydomain.com:8080
debug = yes

DEBUG = False

SECRET_KEY = r'0Afk(,8$cr(Y8:MA""knd>[@$U[G.eQL6DjAmVs'

QUERY_LIMIT = 10000 # maximum number of alerts returned by a single query
HISTORY_LIMIT = 100 #

MongoDB

MONGO_HOST = 'localhost'
MONGO_PORT = 27017
MONGO_DATABASE = 'monitoring'
MONGO_REPLSET = None # 'alerta'

MONGO_USERNAME = 'alerta'
MONGO_PASSWORD = None

AUTH_REQUIRED = False
OAUTH2_CLIENT_ID = 'INSERT-OAUTH2-CLIENT-ID-HERE' # Google or GitHub OAuth2 client ID and secret
OAUTH2_CLIENT_SECRET = 'INSERT-OAUTH2-CLIENT-SECRET-HERE'
ALLOWED_EMAIL_DOMAINS = ['gmail.com']
ALLOWED_GITHUB_ORGS = ['guardian']
API_KEY_EXPIRE_DAYS = 365 # 1 year

CORS_ALLOW_HEADERS = ['Content-Type', 'Authorization', 'Access-Control-Allow-Origin']
CORS_ORIGINS = [
'https://alerta.mydomain.com',
'chrome-extension://jplkjnjaegjgacpfafdopnpnhmobhlaf',
'https://localhost'
]
CORS_SUPPORTS_CREDENTIALS = AUTH_REQUIRED

BLACKOUT_DURATION = 3600 # default period = 1 hour

the concept of 'maintenance mode'?

Does Alerta have a mechanism for the concept of 'maintenance mode'?

ie, I'm going to do work on a server, which would cause alerts, so I want to temporarily mute alerts being generated from it.

Sign up page returns "invalid rounds" error

/cc @menardorama

rpm won't install on systems with prelink

Getting following error when installing RPMs build from alerta.spec file...

# rpm -Uvh packages/alerta/alerta-3.0.1-552.x86_64.rpm
Preparing...                ########################################### [100%]
   1:alerta                 ########################################### [100%]
error: unpacking of archive failed on file /opt/alerta/bin/python;533bfbe3: cpio: Digest mismatch

Running undo prelink during the install step doesn't seem to work.

Work-around to get alerta.rpm to install on systems with prelink use rpm --nomd5 alerta.rpm.

"deploy to heroku" button is not working

There was a problem with the app.json file found at

"https://github.com/guardian/alerta/tree/release/3.2":

No app.json located in the repo URL provided. Are you sure the URL is correct and that an app.json file exists in the project root directory?

Limit history

Is there any way I can call the api/alerts GET method to return the alerts without history?
I understand that one can limit it in the app.config but that would become system wide. I just want it to be request specific.

New alert status wrong for EXPIRED alerts

When alerts are in status expired and a new alert is received for that event-resource combination then the alert stays in the expired status when it should actually change to closed status if normal severity and open status otherwise.

css/js path issues

I installed latest alerta build 3.0.9 in Debian system and installed smoothly. I ran into a issue when accessing from my browser. The css/js files cannot be accessible. This error seems to be odd. I think I am missing something here.

GET /dashboard/index.html HTTP/1.1" 200 -
[09/May/2014 03:31:07] "GET /dashboard/v2/assets/DataTables/media/js/jquery.dataTables.min.js HTTP/1.1" 404 -
[09/May/2014 03:31:07] "GET /dashboard/v2/assets/bootstrap/css/bootstrap.css HTTP/1.1" 404 -
[09/May/2014 03:31:07] "GET /dashboard/v2/assets/bootstrap/css/bootstrap-responsive.css HTTP/1.1" 404 -
[09/May/2014 03:31:07] "GET /dashboard/v2/assets/DataTables/media/js/jquery.js HTTP/1.1" 404 -
[09/May/2014 03:31:07] "GET /dashboard/v2/assets/DataTables/bootstrap/dataTables.bootstrap.css HTTP/1.1" 404 -[09/May/2014 03:31:07] "GET /dashboard/v2/assets/css/custom.css HTTP/1.1" 404 -
[09/May/2014 03:31:07] "GET /dashboard/v2/assets/DataTables/extras/TableTools/media/css/TableTools.css HTTP/1.1" 404 -
[09/May/2014 03:31:08] "GET /dashboard/v2/assets/DataTables/extras/TableTools/media/js/TableTools.min.js HTTP/1.1" 404 -
[09/May/2014 03:31:08] "GET /dashboard/v2/assets/DataTables/bootstrap/dataTables.bootstrap.js HTTP/1.1" 404 -
[09/May/2014 03:31:08] "GET /dashboard/v2/assets/js/config.js HTTP/1.1" 404 -
[09/May/2014 03:31:08] "GET /dashboard/v2/assets/js/console.js HTTP/1.1" 404 -
[09/May/2014 03:31:08] "GET /dashboard/v2/assets/bootstrap/js/bootstrap-collapse.js HTTP/1.1" 404 -
[09/May/2014 03:31:08] "GET /dashboard/v2/assets/bootstrap/js/bootstrap-alert.js HTTP/1.1" 404 -

alert-query message too long for logging debug output

(alerta)nsatterl@GNM40834:~$ alert-query --days 1 --debug
Alerta Report Tool
  api server: monitoring.dc2.gnm:8080
    timezone: Europe/London
    interval: 2014/01/07 11:09:13 - 2014/01/08 11:09:13
     sort by: lastReceiveTime

Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 808, in emit
    self.socket.send(msg)
error: [Errno 90] Message too long
Logged from file api.py, line 124
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 808, in emit
    self.socket.send(msg)
error: [Errno 90] Message too long
Logged from file api.py, line 139

Strange behaviour with prometheus/alertmanager alert transitions

Here is important snippets from my configurations. Prometheus.rules:

ALERT alertName
    IF (metric > 1000) FOR 1m
    LABELS { severity="critical" }
    ANNOTATIONS {
    }

ALERT alertName
    IF (metric > 100) FOR 1m
    LABELS { severity="major" }
    ANNOTATIONS {
    }

Alertmanager.yml:

route:
  receiver:        'alerta'
  group_wait:      10s
  group_interval:  10s
  repeat_interval: 60s
  group_by:        ['alertname', 'host']

inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'major'
  equal: ['alertname', 'host']

receivers:
- name: 'alerta'
  webhook_configs:
  - url: 'http://localhost:8080/webhooks/prometheus?api-key=key'
    send_resolved: true

From alertmanager seems that all is working as expected, but there is a strange behaviour in Alerta.

• If metric rises over 100, the major alarm is firing in alertmanager. Alerta receives the alert correctly.
• If metric rises over 1000 , the critical alarm is firing in alertmanager and the major is inhibited. Alerta updates the alert correctly.
• If metric falls under 1000 again (but is still over 100) alertmnager removes the critical one and the major one is firing again. But Alerta gets confused for some reason - number of duplicates is zeroed, it's sometimes unclear whether its major or critical now, and duplicate updates (different ID, but same datetime etc) will appear under alert data.

Boto Compatibility for Alerta

I am one of the maintainers of Boto. A few weeks ago we began to merge backward-compatible Python 3.3 and 3.4 support into our develop branch. Most modules now support Python 3 with passing unit and integration tests, and I am hoping to do a release soon. I am reaching out to you to ask if you would test our develop branch with Alerta.

Testing against Python 2.x will help to ensure that we don't accidentally break third party modules like Alerta. If you would like, you may also try out the Python 3 support and let us know any feedback. More information about the Python 3 support can be found in this boto-users post:

https://groups.google.com/forum/#!topic/boto-users/vcvoy8zrfP8

I appreciate any time that you can put into helping the Boto project with this.

This request is part of boto/boto#2437

Allow closing alerts on blackout.

An example:

• alert X with severity critical comes,
• a blackout is set cacthing alert X,
• alert X comes with OK (but is neglected).
• blackout is off,
• alert X comes with critical, and becomes a duplicate.

Maybe there should be a blackout (configurable via BLACKOUT_ACCEPT=['ok', 'normal']) that pass alerts of said severities.

Just an idea. (We can provide a PR).

alert-query "watcher" option misses some alerts

updating user (PUT /user/<user>) should update the whole user, not just password

Now only password is updated when you PUT /user/<user>. According to PUT semantics, I expected to update the whole user. There is no alternative way of updating users in alerta than that.

I can provide a PR for this.

API should return 400 error if limit (eg. page size) not greater than zero

It currently returns a 500 internal server error if limit is 0 or a negative number.

Required config undocumented

It is undocumented that in order to properly use alerta web, one needs to add its base uri to alerta server's settings. It is needed even when the URI is the same.

CORS_ORIGINS = [ 
   'http://YOUR_ALERTA_WEB_ADDRESS'
]

Port 8080 is hardwired

I've installed alerta with pip install alerta-server on Mac OS X. Now I'm trying to run an alerta server on a port different from the default 8080, but no luck. After browsing the code I see port 8080 hardwired in this line: https://github.com/guardian/alerta/blob/ced4b0d/alerta/app/__init__.py#L97. Changing this value has the desired effect, but is still hard-wired.

Before I tried setting ALERTA_SVR_CONF_FILE=~/.alertad.conf with the following minimal file content:

[DEFAULT]
timezone = Europe/Berlin
profile = production

[profile production]
endpoint = http://localhost:8090
key = demo-key

But I was getting this traceback:

Flatbook:alerta_test_directory dinu$ ./miniconda2/bin/alertad
Traceback (most recent call last):
  File "./miniconda2/bin/alertad", line 9, in <module>
    load_entry_point('alerta-server==4.7.9', 'console_scripts', 'alertad')()
  File "/Users/dinu/alerta_test_directory/miniconda2/lib/python2.7/site-packages/setuptools-18.8.1-py2.7.egg/pkg_resources/__init__.py", line 568, in load_entry_point
  File "/Users/dinu/alerta_test_directory/miniconda2/lib/python2.7/site-packages/setuptools-18.8.1-py2.7.egg/pkg_resources/__init__.py", line 2720, in load_entry_point
  File "/Users/dinu/alerta_test_directory/miniconda2/lib/python2.7/site-packages/setuptools-18.8.1-py2.7.egg/pkg_resources/__init__.py", line 2380, in load
  File "/Users/dinu/alerta_test_directory/miniconda2/lib/python2.7/site-packages/setuptools-18.8.1-py2.7.egg/pkg_resources/__init__.py", line 2386, in resolve
  File "/Users/dinu/alerta_test_directory/miniconda2/lib/python2.7/site-packages/alerta/app/__init__.py", line 17, in <module>
    app.config.from_envvar('ALERTA_SVR_CONF_FILE', silent=True)
  File "/Users/dinu/alerta_test_directory/miniconda2/lib/python2.7/site-packages/flask/config.py", line 108, in from_envvar
    return self.from_pyfile(rv, silent=silent)
  File "/Users/dinu/alerta_test_directory/miniconda2/lib/python2.7/site-packages/flask/config.py", line 129, in from_pyfile
    exec(compile(config_file.read(), filename, 'exec'), d.__dict__)
  File "/Users/dinu/.alertad.conf", line 5
    [profile production]
                      ^
SyntaxError: invalid syntax

Looking into /Users/dinu/alerta_test_directory/miniconda2/lib/python2.7/site-packages/flask/config.py:129 makes me suspect that alerta expects a Python file.

Is there any released version that works better?

Modifying multiple alerts from the UI

Just posting an enhancement request. I'd like to see a UI enhancement that allows modifying the states of multiple events. For example if a box has multiple services that alert on it, I'd like to ACK them all at once instead of clicking back and forth through the front page and the alert details page.

Alerts on heartbeat - Feature request

The heart beat api in alerta is quite interesting where applications push the status to alerta. But one thing missing is the alerts on heart beats when a particular heart beat becomes stale.

[question] how does alerta api scale?

Can I just add more instances of the API, talking to the same mongodb replica set, to handle more traffic?

I'm thinking of having thousands of hosts sending heartbeats. How many heartbeats does alerta typically handle?

Thanks!

alerta complaining about the amqp transport

2015-06-10 13:23:30,805 - alerta.app[14679]: ERROR - Failed to connect to AMQP transport mongodb://localhost:27017/kombu: Unknown option auto_start_request
2015-06-10 13:23:30,805 - alerta.app[14679]: ERROR - Server plug-in 'amqp' could not be loaded:

do you drive pager alerting via Alerta?

Is there a way to send email based alerting off of alerts? I don't see any plugins for it, and wonder how you guys use this to drive an Ops team.

Basic Auth broken

2015-06-11T00:31:19.977896+00:00 heroku[router]: at=info method=OPTIONS path="/auth/signup" host=alerta-api.herokuapp.com request_id=0783a919-0c66-4231-8d9f-3130e305a890 fwd="109.157.221.123" dyno=web.1 connect=1ms service=4ms status=200 bytes=468
2015-06-11T00:31:20.626010+00:00 heroku[router]: at=info method=POST path="/auth/signup" host=alerta-api.herokuapp.com request_id=9e09b7f9-3a93-4a54-a312-91b3306cc79f fwd="109.157.221.123" dyno=web.1 connect=14ms service=403ms status=500 bytes=568
2015-06-11T00:31:20.625983+00:00 app[web.1]: Traceback (most recent call last):
2015-06-11T00:31:20.625976+00:00 app[web.1]: 2015-06-11 00:31:20,625 - alerta.app[9]: ERROR - Exception on /auth/signup [POST]
2015-06-11T00:31:20.625985+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
2015-06-11T00:31:20.625992+00:00 app[web.1]:     rv = self.handle_user_exception(e)
2015-06-11T00:31:20.625989+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
2015-06-11T00:31:20.625987+00:00 app[web.1]:     response = self.full_dispatch_request()
2015-06-11T00:31:20.625993+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask_cors.py", line 272, in wrapped_function
2015-06-11T00:31:20.625996+00:00 app[web.1]:     return cors_after_request(app.make_response(f(*args, **kwargs)))
2015-06-11T00:31:20.626000+00:00 app[web.1]:     reraise(exc_type, exc_value, tb)
2015-06-11T00:31:20.625998+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
2015-06-11T00:31:20.626001+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
2015-06-11T00:31:20.626003+00:00 app[web.1]:     rv = self.dispatch_request()
2015-06-11T00:31:20.626011+00:00 app[web.1]:   File "/app/alerta/app/auth.py", line 158, in signup
2015-06-11T00:31:20.626014+00:00 app[web.1]:   File "/app/alerta/app/database.py", line 918, in get_user
2015-06-11T00:31:20.626005+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
2015-06-11T00:31:20.626009+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/flask_cors.py", line 152, in wrapped_function
2015-06-11T00:31:20.626010+00:00 app[web.1]:     resp = make_response(f(*args, **kwargs))
2015-06-11T00:31:20.626017+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/collection.py", line 798, in find_one
2015-06-11T00:31:20.626019+00:00 app[web.1]:     for result in cursor.limit(-1):
2015-06-11T00:31:20.626007+00:00 app[web.1]:     return self.view_functions[rule.endpoint](**req.view_args)
2015-06-11T00:31:20.626013+00:00 app[web.1]:     user = db.get_user(user_id)
2015-06-11T00:31:20.626016+00:00 app[web.1]:     user = self.db.users.find_one({"_id": id})
2015-06-11T00:31:20.626026+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/cursor.py", line 813, in __send_message
2015-06-11T00:31:20.626020+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/cursor.py", line 983, in next
2015-06-11T00:31:20.626027+00:00 app[web.1]:     **kwargs)
2015-06-11T00:31:20.626023+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/cursor.py", line 908, in _refresh
2015-06-11T00:31:20.626024+00:00 app[web.1]:     self.__read_preference))
2015-06-11T00:31:20.626029+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/mongo_client.py", line 744, in _send_message_with_response
2015-06-11T00:31:20.626021+00:00 app[web.1]:     if len(self.__data) or self._refresh():
2015-06-11T00:31:20.626040+00:00 app[web.1]: InvalidDocument: Cannot encode object: <pymongo.results.InsertOneResult object at 0x7f3b7ce78140>
2015-06-11T00:31:20.626030+00:00 app[web.1]:     exhaust)
2015-06-11T00:31:20.626032+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/mongo_client.py", line 755, in _reset_on_error
2015-06-11T00:31:20.626035+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/server.py", line 85, in send_message_with_response
2015-06-11T00:31:20.626036+00:00 app[web.1]:     set_slave_okay, sock_info.is_mongos)
2015-06-11T00:31:20.626038+00:00 app[web.1]:     self.ntoreturn, spec, self.fields, self.codec_options)
2015-06-11T00:31:20.626033+00:00 app[web.1]:     return func(*args, **kwargs)
2015-06-11T00:31:20.626037+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/message.py", line 107, in get_message

Use alarm['AlarmName'] instead of alarm['Trigger']['MetricName'] for event in cloudwatch webhook

Motivation:
AWS MetricName (e.g. RequestCount, CPUUtilization, etc.) can be used to define many different CloudWatch Alarms. For example I want alarm when RequestCount is too small on ELB and when it is too high. These two alarm will have the same event name (RequestCount), the same resource (ELB),.

I think using AlertName as an event is better. because it says what is the even.Plus you get to customize it in AWS Cloudwatch.

PS: If you agree I can give PR.

kibana/es integration

Hi,
sorry again if I am asking a silly question, but it's not clear how the kibana/es integration should work.

Let me know if there's a mailing list or some other documentation I should follow to get insights on this.

Really nice system btw, I hope I'll be able to contribute soon (my python is pretty poor, but it's a good opportunity to learn it!)

cheers

Remove urllib as a dependency and replace with Requests

alert-sender and alert-query have already been converted due to proxy issues with OpenShift.

For consistency, we should update all the others:

• alerta/pagerduty/pdclientapi.py
• alerta/common/mail.py
• alerta/urlmon/daemon.py
• alerta/logger/daemon.py
• alerta/ircbot/daemon.py
• alerta/aws/daemon.py
• alerta/solarwinds/swis.py

Raise severities for cloudwatch alarms

Cloudwatch has only two severities: OK and ALARM plu one informative INSUFFICIENT_DATA. Since ALARM is the highest severity I would raise it to critical, INSUFFICIENT_DATA to major and OK leave as normal.

The rationale by this is that since ALARM is used for alarms, it should have the highest priority.

This can be done either statically, just as now or by supplying severty map in config. I can as always provide a PR for both.

[POLICY] Alert environment must be one of Production, Development

Environment name is too restrictive. Instead of only allowing Production or Development, any value should be accepted. For example, we use consul and have several environment names for different consul datacenters. So environment names may be prod-us-region-1, prod-us-region-2, dev-us-region-1, etc.

Monitoring tool for streaming data

Hi,
we are developing a visualization tool for streaming data and anomaly detection at https://github.com/nupic-community/nupic.visualizations I'm wondering if it could be any useful and complementary to your solution?
Thanks for feedback,

[question] what is top10?

Trying to figure out what search top10 is supposed to return?

I have a current alert with a warning status, but top10 returns an empty list.

Angular UI locks Alertad server

If you leave an instance of the Alerta Angular Frontend running, it occasionally locks the alertad server. Any further incoming alerts are not processed by the server, output on stdout or available through the API.

The following is the stack trace output when you keyboard interrupt the running process:

197.228.171.62 - - [18/Feb/2015 05:50:20] "GET /alerts/count HTTP/1.1" 200 -
^C----------------------------------------
Exception happened during processing of request from ('197.228.171.62', 52068)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 568, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 203, in handle
----------------------------------------
Exception happened during processing of request from ('197.228.171.62', 51739)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 568, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 203, in handle
----------------------------------------
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 234, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "/usr/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
Exception happened during processing of request from ('146.148.121.4', 41429)
Traceback (most recent call last):
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
KeyboardInterrupt
----------------------------------------
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 234, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "/usr/lib/python2.7/SocketServer.py", line 555, in process_request
    self.collect_children()
  File "/usr/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/SocketServer.py", line 523, in collect_children
    pid, status = os.waitpid(0, 0)
KeyboardInterrupt
----------------------------------------
KeyboardInterrupt
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('197.228.171.62', 55571)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 568, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 203, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 234, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "/usr/lib/python2.7/socket.py", line 447, in readline
    data = self._sock.recv(self._rbufsize)
KeyboardInterrupt

The IP address in the stack trace above is the public IP of the client accessing the angular UI at the time. So, this is not an issue with posting alerts to the alerta server but rather with getting alerts from the API.

However, hitting ctrl-c doesn't kill the process, but rather seems to kill the lock and any other alerts that have been sent in the interim, now flush through and the process continues.

I'm not sure if this should be raised here or on the Alerta angular UI project.

"Services" seems to be a required field as well

When I post data using API I'm getting 403 "[POLICY] Alert must define a service" so I assume this fields is a required one as well as resource and event are.

Basic auth instead of OAuth

Hi,

Alerta seems to be a great tool and exactly what I'm looking for (Aggregate Zabbix and PingDom)

But we don't want to use external OAuth service and don't want to implement it internally just for alerta.

Is it possible to just have a basic user/password auth ?

As this service won't be exposed on the internet I have no need to couple it to an Exernal Auth mechanism.

Thanks

Can't see the response from Web-UI

Here is my alerta.conf:

DEBUG = True

MONGO_HOST = 'localhost'
MONGO_PORT = 27017
MONGO_DATABASE = 'monitoring'
MONGO_REPLSET = None  # 'alerta'

MONGO_USERNAME = 'alerta'
MONGO_PASSWORD = 'xxxxx'

AUTH_REQUIRED = False

CORS_ALLOW_HEADERS = ['Content-Type', 'Authorization','Access-Control-Allow-Origin']
CORS_ORIGINS = [
    'http://try.alerta.io',
    'http://explorer.alerta.io',
    'chrome-extension://jplkjnjaegjgacpfafdopnpnhmobhlaf',
    'http://localhost'
]
CORS_SUPPORTS_CREDENTIALS = AUTH_REQUIRED

PLUGINS = ['logstash']

AMQP_URL = 'redis://localhost:6379/'              # Redis

LOGSTASH_HOST = 'localhost'
LOGSTASH_PORT = 6379

When run the 'alertad' command, I can see the response is OK, and I also can see the alert records have been already restored into MongoDB.

192.168.x.x - - [08/Jul/2015 13:56:54] "GET /alerts/count?status=open HTTP/1.1" 200 -
192.168.x.x - - [08/Jul/2015 13:56:54] "GET /services?status=open HTTP/1.1" 200 -
192.168.x.x - - [08/Jul/2015 13:56:54] "GET /environments?status=open HTTP/1.1" 200 -
192.168.x.x - - [08/Jul/2015 13:56:54] "GET /alerts?status=open HTTP/1.1" 200 -

I use apache as web server. The alertad and WebUI are in the same server.
But I can't get them from WebUI. Here is the error message by chrome console.
XMLHttpRequest cannot load http://xx.xx.xx.xx:8080/alerts?status=open. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://xx.xx.xx.xx.:9080' is therefore not allowed access.

How can I resolve this issue?

Send alert to external services

Why not use NotifyMe https://github.com/notifymehq for the alert to others services

[enhancements] support configurable OAUTH2 providers

GitLab provides OAuth2 services: http://doc.gitlab.com/ce/integration/oauth_provider.html

If I have a local installation of GitLab, it would be nice to be able to point Alerta at it.

Right now, however, Alerta hard codes the auth endpoints on a per service basis. Could a configurable endpoint be created for OAuth2 sources like this?