gomex / docker-zabbix

Add and monitor running docker containers in Zabbix Server

Goal

Monitor resource used by running containers

TODO

• Get informations about percent used CPU, percent used RAM memory and network
• Add options to get information from all containers and not only using id container
• Create DockerService package
• Write tests for package
• Create zabbix module
• Create python package, to install with pip

Requirements

install all libraries (docker-py and py-zabbix)

# pip install -r requirements.txt

install libraries separately

docker-py library

# pip install docker-py

or

# git clone [email protected]:docker/docker-py.git
# cd docker-py
# python setup.py install

py-zabbix library

# pip install py-zabbix

or

# git clone [email protected]:blacked/py-zabbix.git
# cd py-zabbix
# python setup.py install

Debian problem to use pip

To fix this problem

ImportError: cannot import name 'IncompleteRead'

Reinstall pip package

easy_install -U pip

Packages used

• python 2.7.9
• docker 1.6
• zabbix agent and server 2.4

Initial Configuration

Docker host server

# apt-get install zabbix-agent sudo
# git clone [email protected]:Gomex/docker-zabbix.git
# cd docker-docker
# cp *.py /etc/zabbix
# chown -R zabbix /etc/zabbix/
# chmod u+x /etc/zabbix/*.py
# echo 'Defaults:username !requiretty' >> /etc/sudoers
# echo "zabbix ALL=NOPASSWD: /etc/zabbix/docker_stats.py" >> /etc/sudoers
# echo "EnableRemoteCommands=1" >> /etc/zabbix/zabbix_agentd.conf
# echo "Timeout=30" >> /etc/zabbix/zabbix_agentd.conf
# /etc/init.d/zabbix-agent restart

Zabbix server

1. Import that template
2. Create a host to your Docker host server and link the template "Template Docker Host"
3. Wait the containers :P

If you don't have a zabbix server to test

# docker pull berngp/docker-zabbix
# docker run --name zabbix-server -d -P berngp/docker-zabbix

Check which port was mapped to tcp/80

# docker ps | grep "docker-zabbix"

Check which ip was mapped to new zabbix server container and put in your zabbix-agent conf file

# docker inspect zabbix-server | grep -i ipaddress

Using python3

To use python 3 is necessary install pip for python3

# aptitude install python3-pip

And install its requiriments

# pip3 install -r requirements.txt

Debian problem to use pip

To fix this problem

ImportError: cannot import name 'IncompleteRead'

Reinstall pip package

easy_install3 -U pip

Inspiration

I decided create this solution after read that news and used part of code to start my own.

docker-zabbix's Issues

File "/etc/zabbix/docker_stats.py", line 45, in <module> rx_bytes = stats['networks']['eth2']['rx_bytes']

Bom dia, Estou com o seguinte problema.

[root@localhost zabbix]# /etc/zabbix/docker_stats.py
Traceback (most recent call last):
File "/etc/zabbix/docker_stats.py", line 45, in
rx_bytes = stats['network']['rx_bytes']

Alterei de network para Networks , e rodei novamente e deu erro na proxima linha

Traceback (most recent call last):
File "/etc/zabbix/docker_stats.py", line 46, in
tx_bytes = stats['networks']['tx_bytes']
KeyError: 'tx_bytes'

, troquei todos e até inseri a interface, que ele pede. mas apresenta esse erro

[root@localhost zabbix]# /etc/zabbix/docker_stats.py
Traceback (most recent call last):
File "/etc/zabbix/docker_stats.py", line 45, in
rx_bytes = stats['networks']['eth2']['rx_bytes']
KeyError: 'networks'

ativei o print stats. sai isso

[root@localhost zabbix]# /etc/zabbix/docker_stats.py
{u'blkio_stats': {u'io_service_time_recursive': [], u'sectors_recursive': [], u'io_service_bytes_recursive': [], u'io_serviced_recursive': [], u'io_time_recursive': [], u'io_queue_recursive': [], u'io_merged_recursive': [], u'io_wait_time_recursive': []}, u'precpu_stats': {u'cpu_usage': {u'total_usage': 0, u'usage_in_kernelmode': 0, u'usage_in_usermode': 0}, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}, u'name': u'/stack_microservice-sale.1.v0wo9thnlcuos5c78nqd6ujud', u'read': u'2018-02-10T14:49:14.664550913Z', u'storage_stats': {}, u'id': u'053e0c8370755e76b346ccb42bdf067b3c8c8939e00c6b662b411fdb1326f876', u'num_procs': 0, u'preread': u'0001-01-01T00:00:00Z', u'memory_stats': {u'usage': 1090748416, u'stats': {u'unevictable': 0, u'total_inactive_file': 12288, u'total_rss_huge': 1023410176, u'hierarchical_memsw_limit': 9223372036854771712, u'total_cache': 139264, u'total_mapped_file': 32768, u'mapped_file': 32768, u'pgfault': 460124, u'total_writeback': 0, u'hierarchical_memory_limit': 9223372036854771712, u'total_active_file': 126976, u'rss_huge': 1023410176, u'cache': 139264, u'active_anon': 1090609152, u'pgmajfault': 0, u'total_pgpgout': 145274, u'pgpgout': 145274, u'inactive_file': 12288, u'total_unevictable': 0, u'total_pgfault': 460124, u'total_pgmajfault': 0, u'writeback': 0, u'total_active_anon': 1090609152, u'pgpgin': 156581, u'total_dirty': 0, u'total_pgpgin': 156581, u'rss': 1090609152, u'total_inactive_anon': 0, u'inactive_anon': 0, u'active_file': 126976, u'dirty': 0, u'total_rss': 1090609152}, u'limit': 135014055936, u'max_usage': 1134514176}, u'pids_stats': {u'current': 126}, u'networks': {u'eth2': {u'tx_dropped': 0, u'rx_packets': 6865, u'rx_bytes': 586994, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 585489, u'rx_dropped': 0, u'tx_packets': 8282}, u'eth1': {u'tx_dropped': 0, u'rx_packets': 2755, u'rx_bytes': 670879, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 627063, u'rx_dropped': 0, u'tx_packets': 5065}, u'eth0': {u'tx_dropped': 0, u'rx_packets': 141, u'rx_bytes': 15888, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 76157, u'rx_dropped': 0, u'tx_packets': 137}}, u'cpu_stats': {u'cpu_usage': {u'total_usage': 132888729266, u'usage_in_usermode': 90960000000, u'usage_in_kernelmode': 28260000000, u'percpu_usage': [11345977579, 7422004586, 11328669783, 8138081001, 13561496515, 7584799570, 10418815755, 5341014566, 12428419977, 7222476591, 10909197605, 7182642230, 2193149273, 1044650204, 2917885223, 2023724417, 1281495879, 1153041574, 1561230391, 1587410219, 1515778038, 1230465915, 1898256543, 1598045832]}, u'system_cpu_usage': 4449007650000000, u'online_cpus': 24, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}}
{u'blkio_stats': {u'io_service_time_recursive': [], u'sectors_recursive': [], u'io_service_bytes_recursive': [], u'io_serviced_recursive': [], u'io_time_recursive': [], u'io_queue_recursive': [], u'io_merged_recursive': [], u'io_wait_time_recursive': []}, u'precpu_stats': {u'cpu_usage': {u'total_usage': 0, u'usage_in_kernelmode': 0, u'usage_in_usermode': 0}, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}, u'name': u'/stack_microservice-product.1.l0vtgjjzufz6cki9jdhnafxp2', u'read': u'2018-02-10T14:49:17.664457855Z', u'storage_stats': {}, u'id': u'b69b66271553afafbaf37d2082f3c6cb018599ee679f709a0885d9de07e5d1af', u'num_procs': 0, u'preread': u'0001-01-01T00:00:00Z', u'memory_stats': {u'usage': 1094557696, u'stats': {u'unevictable': 0, u'total_inactive_file': 12288, u'total_rss_huge': 1035993088, u'hierarchical_memsw_limit': 9223372036854771712, u'total_cache': 69632, u'total_mapped_file': 32768, u'mapped_file': 32768, u'pgfault': 352700, u'total_writeback': 0, u'hierarchical_memory_limit': 9223372036854771712, u'total_active_file': 57344, u'rss_huge': 1035993088, u'cache': 69632, u'active_anon': 1094488064, u'pgmajfault': 0, u'total_pgpgout': 136926, u'pgpgout': 136926, u'inactive_file': 12288, u'total_unevictable': 0, u'total_pgfault': 352700, u'total_pgmajfault': 0, u'writeback': 0, u'total_active_anon': 1094488064, u'pgpgin': 149163, u'total_dirty': 0, u'total_pgpgin': 149163, u'rss': 1094488064, u'total_inactive_anon': 0, u'inactive_anon': 0, u'active_file': 57344, u'dirty': 0, u'total_rss': 1094488064}, u'limit': 135014055936, u'max_usage': 1131569152}, u'pids_stats': {u'current': 89}, u'networks': {u'eth2': {u'tx_dropped': 0, u'rx_packets': 9232, u'rx_bytes': 718189, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 702045, u'rx_dropped': 0, u'tx_packets': 9543}, u'eth1': {u'tx_dropped': 0, u'rx_packets': 2837, u'rx_bytes': 629849, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 592040, u'rx_dropped': 0, u'tx_packets': 5110}, u'eth0': {u'tx_dropped': 0, u'rx_packets': 26, u'rx_bytes': 2502, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 7592, u'rx_dropped': 0, u'tx_packets': 14}}, u'cpu_stats': {u'cpu_usage': {u'total_usage': 103421698042, u'usage_in_usermode': 68600000000, u'usage_in_kernelmode': 23630000000, u'percpu_usage': [3868059131, 12169660783, 4283518186, 11977762882, 5364689356, 11951454754, 4998659787, 8899396433, 4783902124, 8661666404, 4841360238, 8663620995, 1352963279, 368967177, 2391290194, 981160827, 551950448, 936148253, 915911869, 907800544, 923314650, 1360116612, 1045573324, 1222749792]}, u'system_cpu_usage': 4449079570000000, u'online_cpus': 24, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}}
{u'blkio_stats': {u'io_service_time_recursive': [], u'sectors_recursive': [], u'io_service_bytes_recursive': [], u'io_serviced_recursive': [], u'io_time_recursive': [], u'io_queue_recursive': [], u'io_merged_recursive': [], u'io_wait_time_recursive': []}, u'precpu_stats': {u'cpu_usage': {u'total_usage': 0, u'usage_in_kernelmode': 0, u'usage_in_usermode': 0}, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}, u'name': u'/stack_microservice-notification.1.lilz1jv0kkvshnrf84zls7iv0', u'read': u'2018-02-10T14:49:20.664464752Z', u'storage_stats': {}, u'id': u'c2643a9a6081050d6289fd23eb5f20c9253bfee5b8bd05f84fe0713bed13d902', u'num_procs': 0, u'preread': u'0001-01-01T00:00:00Z', u'memory_stats': {u'usage': 820445184, u'stats': {u'unevictable': 0, u'total_inactive_file': 12288, u'total_rss_huge': 754974720, u'hierarchical_memsw_limit': 9223372036854771712, u'total_cache': 40960, u'total_mapped_file': 32768, u'mapped_file': 32768, u'pgfault': 280356, u'total_writeback': 0, u'hierarchical_memory_limit': 9223372036854771712, u'total_active_file': 28672, u'rss_huge': 754974720, u'cache': 40960, u'active_anon': 820404224, u'pgmajfault': 0, u'total_pgpgout': 109595, u'pgpgout': 109595, u'inactive_file': 12288, u'total_unevictable': 0, u'total_pgfault': 280356, u'total_pgmajfault': 0, u'writeback': 0, u'total_active_anon': 820404224, u'pgpgin': 113164, u'total_dirty': 0, u'total_pgpgin': 113164, u'rss': 820404224, u'total_inactive_anon': 0, u'inactive_anon': 0, u'active_file': 28672, u'dirty': 0, u'total_rss': 820404224}, u'limit': 135014055936, u'max_usage': 823812096}, u'pids_stats': {u'current': 67}, u'networks': {u'eth2': {u'tx_dropped': 0, u'rx_packets': 62471, u'rx_bytes': 8080592, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 6798417, u'rx_dropped': 0, u'tx_packets': 86145}, u'eth1': {u'tx_dropped': 0, u'rx_packets': 3810, u'rx_bytes': 261596, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 260864, u'rx_dropped': 0, u'tx_packets': 3800}, u'eth0': {u'tx_dropped': 0, u'rx_packets': 12, u'rx_bytes': 744, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 0, u'rx_dropped': 0, u'tx_packets': 0}}, u'cpu_stats': {u'cpu_usage': {u'total_usage': 196139408094, u'usage_in_usermode': 151410000000, u'usage_in_kernelmode': 41370000000, u'percpu_usage': [5660214816, 14776094139, 5480913941, 16964455905, 25831293696, 17921483861, 8371275606, 12006557168, 6985024879, 14519357140, 6320085742, 15386496066, 1304581838, 1284600037, 27584231161, 2994027016, 2300718125, 778389871, 875600876, 1337899367, 1362260307, 1046877878, 3750191958, 1296776701]}, u'system_cpu_usage': 4449151460000000, u'online_cpus': 24, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}}
Traceback (most recent call last):
File "/etc/zabbix/docker_stats.py", line 45, in
rx_bytes = stats['networks']['eth2']['rx_bytes']
KeyError: 'networks'

Pode me dar uma Luz, já não sei o que fazer para funcionar...

Improve check stats speed

The check spend almost 12 seconds to get all information. We need to improve it.

Create python package, to install with pip

We need create a package to facilitate installation.

Value should be a JSON object.

HI, I have on zabbix server i have created hosto to docker with;
Template Docker Host: Discover docker coontainers key = system.run[/etc/zabbix/docker_stats.py]
and have error (Value should be a JSON object).

On docker host, i run scripts /etc/zabbix/docker_stats.py without error.

Error

Hi,

I am getting the following error message

/usr/bin/sudo /etc/zabbix/docker_stats.py
Traceback (most recent call last):
File "/etc/zabbix/docker_stats.py", line 4, in
from docker_service import DockerService
ImportError: No module named docker_service

Please help
Ferdous Shibly

Remove cron configuration

The dependency for configure docker host with cron file is annoying and not smart. We need find a way to get Docker information without that access configuration.

Python3 compatibility

Optimization

Optimization and simplification of code

Erro (key_pkt_sent) - (Com solução)

Há uma falha ao enviar o resultado do script para o servidor Zabbix devido a um erro na chave key_pkt_sent, linha 39.

Saída do erro: ./docker_stats.py
{"failed": 1, "chunk": 1, "total": 10, "processed": 9, "time": "0.000265"}

Solução: Alterar a linha 39 do arquivo docker_stats.py conforme abaixo:

De: key_pkt_sent = 'user.docker[packets sent]'
Para: key_pkt_sent = 'user.docker[packets_sent]'

README update

It seems to me that in the current implementation of the file README it is probably redundant. There is no need to describe things like install zabbix-agent. If a person is interested in the monitoring of containers it is unlikely that he needs to explain such simple concepts. My plan is to add a translation to Russian, not sure but it may be in German.

Compatible with PEP8

I believe that the final code should be tested for compliance with PEP8.

Missing file

In this repository there is not enough file list_containers_docker_lld.py and a link to the installation instructions leads to gitlab.

Remove sudo access

The dependency for configure docker host with sudo access is annoying and not smart. We need find a way to get Docker information without that access configuration.

Write tests for package

We need test. It is really important.