When one of my docker container fail I get the alert as much time as my number of containers about alertmanager

[juliusv]

Could you share your alert.rules?

"/metrics" is the standard endpoint for Prometheus to scrape metrics from
an instrumented job - could you explain what part of that seems strange to
you?

What do you mean, you are using cadvisor for "prometheus analytics", when
you say the scraped targets are all sinatra apps? Could it be that you
meant to scrape cAdvisor, but you are accidentally scraping the containers
directly (which would be correct if you instrumented them correctly with
Prometheus metrics)?

By the way, you don't need the "labels: job: '...'" sections in your
config, since the job label will default to what you set in the "job_name"
already, which has the same value in your config.

On Tue, Oct 27, 2015 at 10:22 PM, casertap [email protected] wrote:

here is my prometheus.yml

scrape_configs:
- job_name: 'streamapp'
scrape_interval: 5s
scrape_timeout: 10s
target_groups:
- targets: ['streamapp-container:3002']
labels:
job: 'streamapp'
- job_name: 'dashboardapp'
scrape_interval: 5s
scrape_timeout: 10s
target_groups:
- targets: ['dashboardapp-container:3000']
labels:
job: 'dashboardapp'
- job_name: 'analyzeapp'
scrape_interval: 5s
scrape_timeout: 10s
target_groups:
- targets: ['analyzeapp-container:3001']
labels:
job: 'analyzeapp'

All the containers are sinatra apps. I have setup '/metrics' routes on
each service to have alertmanager behave correctly.
If I do not provide a '/metrics' route my container is detected has dead.
Everything looks ok but when I stop one container (let say dashboardapp).
I get all 3 alerts in red and received 3 msg via slack. The 3 messages are
saying that dashboardapp is down.

Am I doing things right?
I found it strange to have to provide a '/metrics' route and the fact that
I received 3 times the same message is strange.

—
Reply to this email directly or view it on GitHub
#139.

[casertap]

I give you my alert.rules

### alert.rules ###

# Alert for any instance that is unreachable for >5 minutes.
ALERT streamapp
    IF up == 0
    FOR 10s
    SUMMARY "Instance {{$labels.instance}} down"
    DESCRIPTION "{{$labels.instance}} of job {{$labels.job}} has been down for more than 10s"

ALERT dashboardapp
    IF up == 0
    FOR 10s
    SUMMARY "Instance {{$labels.instance}} down"
    DESCRIPTION "{{$labels.instance}} of job {{$labels.job}} has been down for more than 10s"

ALERT analyzeapp
    IF up == 0
    FOR 10s
    SUMMARY "Instance {{$labels.instance}} down"
    DESCRIPTION "{{$labels.instance}} of job {{$labels.job}} has been down for more than 10s"

And here is my docker-compose (so you can better understand my architecture

datadb:
  image: busybox
  container_name: datadb
  volumes:
    - /data/db

mongo:
  image: mongo
  container_name: mongo
  volumes_from:
    - datadb
  ports:
    - 27017:27017

streamapp:
  build: streamapp/
  container_name: streamapp-container
  links:
    - mongo
  ports:
    - 3002:3002
  expose:
    - 3002
  env_file:
    - './.env'

dashboardapp:
  build: dashboardapp/
  container_name: dashboardapp-container
  links:
    - mongo
  ports:
    - 3000:3000
  expose:
    - 3000
  env_file:
    - './.env'


analyzeapp:
  build: analyzeapp/
  container_name: analyzeapp-container
  ports:
    - 3001:3001
  expose:
    - 3001
  links:
    - mongo
  env_file:
    - './.env'


cadvisor:
  image: google/cadvisor
  container_name: cadvisor
  ports:
    - "8080:8080"
  volumes:
    - "/:/rootfs:ro"
    - "/var/run:/var/run:rw"
    - "/sys:/sys:ro"
    - "/var/lib/docker/:/var/lib/docker:ro"

promalertmgr:
  build: promalertmgr/
  container_name: promalertmgr
  ports:
    - "9093:9093"
  expose:
    - "9093"
  command: "-config.file=/alertmanager.conf"
  env_file:
    - './.env'

prometheus:
  build: prometheus/
  container_name: prometheus
  links:
    - cadvisor
    - promalertmgr
    - streamapp
    - dashboardapp
    - analyzeapp
    - mongo
  ports:
    - "9090:9090"
  command: "-config.file=/etc/prometheus/prometheus.yml -alertmanager.url=http://192.168.99.100:9093"
  env_file:
    - './.env'

Prometheus works fine when I connect it to cadvisor:

scrape_configs:                         
    - job_name: 'cadvisor'             
      scrape_interval: 5s               
      scrape_timeout: 10s               
      target_groups:                    
        - targets: ['cadvisor:8080']    

But what I want is monitor my streamapp, dashboardapp, and analyzeapp container and receive an alert when one of them is down.

If I connect prometheus to cadvisor like I did and add the 3 rules above. Those rules stay green even if I stop the dashboard container.
If I add "/metrics" end point to my apps and connect prometheus to each app while I did in my first post. If I stop one container I get alerted. But the metric do not work because I did not connect cadvisor.

How can I have the metric with cadvisor and detect if one of my container is down?

Thanks for your answer.
I hope to hear from you soon.

[casertap]

Ok so what I needed was something like this

ALERT streamapp
    IF absent(container_last_seen{name="streamapp-container"})
    SUMMARY "Instance {{$labels.instance}} down"
    DESCRIPTION "{{$labels.instance}} of job {{$labels.job}} has been down for more than 10s"

I only need to connect prometheus to cadvisor and than use a cadvisor metric to detect if the container is down.

It takes a lot of time to detect the app is down but still, it is working.
Thanks.

[casertap]

Now that I have updated the docker image it does not work anymore.
I need to do it like this:

ALERT wpprog
    IF absent(((time() - container_last_seen{name="wpprog"}) < 5)) FOR 15s
    SUMMARY "Instance is down"
    DESCRIPTION "You should do something"

This is working fine.

[fabxc]

This seems to be resolved.

[Shashidhar1121]

hi castertap
can u post the dockerfile for above prometheus version