add to prometheus about go-mux

[utf18]

hey,

i guess the issue is that the http.ListenAndServe function is blocking and your doSomeWork() is never called.
i went around this by calling a goroutine which runs before the http server is started and is running concurrently.

i did not read it correctly. my bad i will have a look at your code and come back to you.
Maybe you have to register the counter in the function

i added some code covering this, hope this helps.
f0d5b68

[Arnold1]

the endpoint http://localhost:8080/metrics shows me the following:

# HELP hello_requests_total Total number of /hello requests.
# TYPE hello_requests_total counter
hello_requests_total{status="200"} 1
hello_requests_total{status="400"} 2
hello_requests_total{status="500"} 1

but i cannot see any metrics like that in prometheus...do you know what i mean? do you see the same issue?

[Arnold1]

in the prometheus ui http://localhost:9090/graph i only see the following metrics to select:

go_
http_
net_
process_
prometheus_
scrape_
tsdb_
up

here is what i do:

$ CGO_ENABLED=0 GOOS=darwin go build -o mux
$ ./mux

$ docker run -p 9090:9090 -v `pwd`/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

[utf18]

i think i know what's happening.
this setup works under the assumption that you run it with docker-compose. If you run it as you describe the "target" in the prometheus.yml will not exist.

this is due to the reason, that the docker-compose uses a docker network to make use of internal DNS, so the go-mux-example described in the compose file is actually a valid dns name inside of the compose network.

i don't think your setup will work, because docker is not intended to connect from inside the container to the outer host.

You could try with --net=host, but i would recommend that you either run mux in a container oder run the prometheus outside of the container to test your changes.

you can build a mux docker image with:
(assuming that you build it outside with GOOS=linux)

FROM golang:alpine
ADD .mux /mux
RUN chmod 0755 /mux
EXPORT 8080
CMD ["/mux"]

and then build it with
docker build -t mux:v1 .

after that you can change my image in the compose file with mux:v1 and everything should work fine or start them both inside of a container on your own.

tl:dr if your metrics are working in the /metrics endpoint you will have to take a hard look on prometheus in the /targets.
should look like this:

[Arnold1]

works now:

my docker file:

FROM golang:1.9.2

RUN apt-get update && apt-get install -y git

RUN mkdir -p /home/go-mux-example
WORKDIR /home/go-mux-example

COPY main.go /home/go-mux-example
COPY handlers.go /home/go-mux-example

RUN go get -v github.com/gorilla/handlers
RUN go get -v github.com/gorilla/mux
RUN go get -v github.com/prometheus/client_golang/prometheus
RUN go get -v github.com/sirupsen/logrus

RUN chmod 0755 /home/go-mux-example
RUN CGO_ENABLED=0 GOOS=linux go build -o mux

EXPOSE 8080
CMD ["/home/go-mux-example/mux"]

and then build the docker with
docker build -t mux:v1 .

prometheus.yml file:

# my global config
global:
    scrape_interval:     15s
    evaluation_interval: 15s

    external_labels:
        monitor: 'myapp-monitor'

rule_files:
    # - "first.rules"
    # - "second.rules"

scrape_configs:
    - job_name: 'prometheus'
      static_configs:
          - targets: ['localhost:9090']

    - job_name: 'mux'
      static_configs:
          - targets: ['go-mux-example:8080']

docker compose file:

version: "2"

services:

  prometheus-server:
    image: prom/prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana
    ports:
      - 3000:3000
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=password
    links:
      - prometheus-server:prometheus

  go-mux-example:
    image: mux:v1
    ports:
      - 8080:8080

than start docker compose:
docker-compose up

here is what i see:

ok i can see the inc_value and hello_requests_total in here: http://localhost:9090/graph

[Arnold1]

do you know how to add alerts to prometheus which sends out an email and hipchat msg? for example: if counter reaches 50 -> send alert

[utf18]

nice 👍

for the alerting part:
you have to have the alertmanager up and running:
https://prometheus.io/docs/alerting/configuration/#<email_config>
https://github.com/prometheus/alertmanager
and for the rules alerting the events you want to trigger:
https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/

or else get yourself started with the maybe simpler alerting system from grafana itself and alert on charts.
http://docs.grafana.org/alerting/notifications/

if i have the time, i can add a basic email alerting setup with a local mailserver or webhook to demonstrate, but for now the alerting depends on what you want to do.

i would suggest you add the alertmanager to the compose file and play with it a little bit.

Feel free to close the issue, if this covers your questions.

all the best,

benjamin