technofy / cloudwatch_exporter Goto Github PK

sudo go get github.com/percona/rds_exporter
cannot find package "github.com/prometheus/common/log" in any of:
/usr/lib/golang/src/github.com/prometheus/common/log (from $GOROOT)
/root/go/src/github.com/prometheus/common/log (from $GOPATH)

Please add support for aws_extended_statistics as done in prometheus/cloudwatch_exporter#57

We're trying to get the TargetResponseTime metrics as described in the docs and as noted:

Statistics: The most useful statistics are Average and pNN.NN (percentiles).

So I tried:

      - aws_namespace: 'AWS/ApplicationELB'
        aws_metric_name: 'TargetResponseTime'
        aws_statistics: ['p90', 'p95', 'p99']
        aws_dimensions: ['TargetGroup', 'LoadBalancer']
        aws_dimensions_select_regex:
          TargetGroup: .*
          LoadBalancer: app/production-.*

and just aws_statistics: ['p95']:

      - aws_namespace: 'AWS/ApplicationELB'
        aws_metric_name: 'TargetResponseTime'
        aws_statistics: ['p95']
        aws_dimensions: ['TargetGroup', 'LoadBalancer']
        aws_dimensions_select_regex:
          TargetGroup: .*
          LoadBalancer: app/production-.*

But both returns this error in the logs:

Oct 09 04:32:14 REDACTED cloudwatch_exporter[13161]: InvalidParameterValue: The parameter Statistics.member.1 must be a value in the set [ Minimum, Maximum, Average, SampleCount, Sum ].
Oct 09 04:32:14 REDACTED cloudwatch_exporter[13161]:         status code: 400, request id: REDACTED

Hello,

I have issue to scrape the data from the prometheus, I keep getting the following errors in prometheus :

level=debug ts=2018-06-22T13:18:17.008625038Z caller=scrape.go:703 component="scrape manager" scrape_pool=ec2_cloudwatch target="http://127.0.0.1:9042/scrape?region=eu-west-1&target=i-0123456598549&task=ec2_cloudwatch" msg="Scrape failed" err="context deadline exceeded"

you will find attach my configuration for the cloudwatch_exporter and the prometheus.

Regarding the cloudwatch_exporter I used go get to install it and for prometheus I took the latest version 2.3.1

I tried to increase the scrape time but it did not help.

Thank you by advance .

config-cloudwatchexporter.txt
prometheus-config.txt

Something that might be worth making clear in the documentation: the aws dimension select in this tool is called aws_dimensions_select_regex in the config file, whereas in the official Java tool, it's called aws_dimension_select_regex (ie without the "s").

This had me scratching my head for a while when moving a config over from the official exporter to this one.

Also, the aws_dimensions_select_regex in this tool only supports 1 regexp per dimension, whereas the official one supports a list. It might be worth considering adding support for a list of reg exps as a new key called aws_dimension_select_regex and then it would be compatible with the official tool in both key name and format?

You could leave the existing aws_dimensions_select_regex supporting a single regexp for backwards compatibility perhaps.

I thought i'd open a separate issue to not hijack my previous ticket, sooo...

I haven't actually double-checked, but i think with the "official" cloudwatch-exporter it's possible to omit aws_dimension_select and just get all metrics for a specific dimension.
The config for this would look like the following:

---
tasks:
- name: vpn
  metrics:
  - aws_namespace: "AWS/VPN"
    aws_metric_name: TunnelState
    aws_dimensions: [VpnId]
    aws_statistics: [Average]

Unfortunately this crashes the exporter:

CloudWatch exporter started...
InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, GetMetricStatisticsInput.Dimensions[0].Value.

panic: inconsistent label cardinality

goroutine 349 [running]:
github.com/Technofy/cloudwatch_exporter/vendor/github.com/prometheus/client_golang/prometheus.MustNewConstMetric(0xc420139490, 0x2, 0x3fe0000000000000, 0xc420316920, 0x1, 0x2, 0x0, 0x1)
	/home/jan/go/src/github.com/Technofy/cloudwatch_exporter/vendor/github.com/prometheus/client_golang/prometheus/value.go:172 +0xb0
main.scrape(0xc420326c80, 0xc4205689c0)
	/home/jan/go/src/github.com/Technofy/cloudwatch_exporter/aws.go:100 +0x1047
main.(*cwCollector).Collect(0xc420326c80, 0xc4205689c0)
	/home/jan/go/src/github.com/Technofy/cloudwatch_exporter/collector.go:111 +0x59
github.com/Technofy/cloudwatch_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc420313840, 0xc4205689c0, 0x9dc1e0, 0xc420326c80)
	/home/jan/go/src/github.com/Technofy/cloudwatch_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x61
created by github.com/Technofy/cloudwatch_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/home/jan/go/src/github.com/Technofy/cloudwatch_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:434 +0x2ec

When using variable for role_arn no metrics are coming.

Below are the details:

1.) prometheus config:

• job_name: 'sample_redshift'
  metrics_path: '/scrape'
  ec2_sd_configs:
  • region: us-east-2
    static_configs:
  • targets: ['arn:aws:iam::xxxxxxxxxxxx:role/cloudwatch-role']
    relabel_configs:
  • source_labels: [address]
    target_label: __param_targetrole
  • target_label: address
    replacement: 'localhost:9042'

2.) Target Url:
curl 'http://localhost:9042/metrics?targetrole=arn%3Aaws%3Aiam%3A%3Axxxxxxxxxxxxxx%3Arole%2Fcloudwatch-role&region=us-east-2'

3.) Ouptut from exporter:
HELP cloudwatch_requests_total API requests made to CloudWatch
TYPE cloudwatch_requests_total counter
cloudwatch_requests_total 0

4.) sample.yml config:

tasks:

• name: redshift_cloudwatch
  default_region: us-east-2
  role_arn: [$_targetrole]
  metrics:
  • aws_namespace: AWS/Redshift
    aws_metric_name: DatabaseConnections
    aws_dimensions: [ClusterIdentifier]
    aws_statistics: [Average]

5.) from aws-cli everything is working fine when switching between the roles.

Hi, could you please provide a versioned/tagged image via the official Docker hub?

Thanks in advance!

If a config like the following is used:

  metrics:
      - aws_namespace: 'AWS/ECS'
        aws_metric_name: 'MemoryUtilization'
        aws_dimensions: [ 'ClusterName', 'ServiceName' ]
        aws_statistics: [ 'Maximum' ]
        aws_dimensions_select_regex:
            ServiceName: 'bot-.*'
            ClusterName: 'us-east-1-prod-public'

Then it produces metrics with the "service_name" and "cluster_name" values swapped over, e.g. :

aws_ecs_memory_utilization_maximum{cluster_name="bot-blah",service_name="us-east-1-prod-public",task="bots-dev"} 8.69140625

If the aws_dimensions entries are swapped around, to be [ 'ServiceName', 'ClusterName' ] then the dimensions become correctly labelled.

aws_ecs_memory_utilization_maximum{cluster_name="us-east-1-prod-public",service_name="bot-blah",task="bots-dev"} 8.69140625

Is there a helm chart available?

I'm wondering if you are able to start tagging releases with patch notes so we can download specific versions of this software. It's a bit difficult to download whatever's in master if something significant changes.

While you will generally be OK to retrieve a metric with the 3 parameters passed in by Prometheus aka task+region+target, there is a case where you need a 'target2' or equivalent.

If you use elastic search then the corresponding metrics are a two-dimension one. ClientId+DomainName.
The ClientId is in fact your AWS account ID (not changing and the domain is your choice when you create an ES domain and you can have many of them.
We have aws_dimensions: [ClientId, Domain].
The problem is that the account ID will be different in your DEV environment, STAGE environment, PROD environment... So you cannot hard-code it in the Docker image. At the same time, you may want to get the metrics for multiple domains.

So there is no simple solution even with the regexp support (thanks for it!!!) of #4 and #6. If you want a 'portable' container that you can deploy in any environment of yours with flexible options, you want target and 'target2' as parameters received from Prometheus (if this is even possible).

The support for aws_dimensions_select and aws_dimensions_select_regex has still some constraints, e.g. you cannot mix a aws_dimensions_select value (target) for the domain and a aws_dimensions_select_regex ([0-9]+) for the clientid which could have done the trick (cost of an extra AWS call on each exporter call to the list of values when you use a regexp or no value at all (interpreted as .*).
Hard-coding a list of values like

aws_dimensions_select:
  ClientId: [1234, 4654, 9545]

(the IDs of your accounts) does not work either with on the next line Domain: [$_target] and in any case we would make more AWS calls than needed.

The only solution that I have found, trying many ones is to declare the two-dimension metric and to not specify any value. It work but because of the price is 1 AWS call to get a list of client IDs (only one is returned) plus 1 AWS call to get a list of domains (let say returns N domains). Then you make 1xN individual calls so a total of 2+N AWS calls.
If you could pass target + 'target2' you would make 1 AWS call.

Other metrics may have the same problem but this one was one that I required to get.
I think that it is a good case for a 'target2' parameter if this is doable.

Thanks for this project.

Adding a metric with more than one value in aws_statistics throws error:

* collected metric "aws_lambda_duration_maximum" { label:<name:"function_name" value:"LambdaFunctionName" > label:<name:"task" value:"lambda_duration_no_select" > gauge:<value:290.72 > } was collected before with the same name and label values

 - name: lambda_duration_no_select
   default_region: us-east-1
   metrics:
    - aws_namespace: "AWS/Lambda"
      aws_dimensions: [FunctionName]
      aws_metric_name: Duration
      aws_statistics: [Maximum, Minimum]

We are using OracelDB_exporter with the CUSTOM metrics and we are migrating into RDS now and have few questions. If I use this, can we have that option?.

Example:- https://github.com/iamseth/oracledb_exporter/blob/master/custom-metrics-example/custom-metrics.toml