molu8bits / s3bucket_exporter Goto Github PK

While i have filled the docker-compuose.yml with wasabi endpoint, etc.
When i try to run it, i am given the following error:

ysername@servername:~/s3bucket_exporter-master$ sudo docker-compose up
ERROR: The Compose file './docker-compose.yaml' is invalid because:
services.s3bucketexporter.environment.S3_DISABLE_SSL contains false, which is an invalid type, it should be a string, number, or a null

Hi,
I don't manage to make it work on a s3 compatible storage that is not AWS.
On my local computer on windows 10, docker desktop and Git Bash, I do :

docker run -p 9655:9655 -d -e LISTEN_PORT=:9655 -e S3_DISABLE_SSL=False -e S3_ENDPOINT=https://s3.waw2-1.cloudferro.com -e S3_ACCESS_KEY=xxxxxxxxx -e S3_SECRET_KEY=xxxxxxxxxxxxxxxx -e S3_NAME=s3_cloudferro -e S3_DISABLE_ENDPOINT_HOST_PREFIX=True -e LOG_LEVEL=Debug -e S3_FORCE_PATH_STYLE=True docker.io/molu8bits/s3bucket_exporter:1.0.2

Then I do:
curl -v http://127.0.0.1:9655/metrics

I get:

*   Trying 127.0.0.1:9655...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 9655 (#0)
> GET /metrics HTTP/1.1
> Host: 127.0.0.1:9655
> User-Agent: curl/7.84.0
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:--  0:07:56 --:--:--     0

In the logs I have:

time="2024-04-12T10:13:08Z" level=info msg="Beginning to serve on port:9655"
time="2024-04-12T10:13:08Z" level=info msg="s3 name 's3_wekeo' available at s3 endpoint 'https://s3.waw2-1.cloudferro.com' will be monitored"
time="2024-04-12T10:13:08Z" level=info msg="listenPort ::9655"

In the docker statistics, after one minute or so, the CPU usage increases and the network input increases by ~3 Mb/sec, reaching 500 Mb after like 2-3 minutes.

S3_NAME is not supposed to match with something, right ? It's just a string that I choose.
Do you see any mistake ?

Thank you

Counts only max 1000 objects per bucket

Hello @molu8bits . you said in the readme that this project was initially made to work with Ceph. im currently trying to use it for it however i kept getting empty information from my setup:

level=debug msg="s3metrics read from s3_endpoint :{Test-bucket2 true 0 0 []}"

im wondering if you know what type of capabilities the user needs to have in order to get the right information about the buckets.

also, i dont know what should be the default values for this parameters to work with ceph:
S3_DISABLE_ENDPOINT_HOST_PREFIX
S3_FORCE_PATH_STYLE

kind regards

What if you want to monitor multiple buckets spread over multiple accounts. Is there a way to do this without having to depoly multiple Dockers?

If I knew how I would submit a PR for the README because this is really hard to figure out for someone who doesn't program and just wants to monitor s3 buckets.

I've tried all sorts of combinations I found in various docs like this one and this one.

I get different errors depending on what url I try but here's on example using S3_ENDPOINT=https://bucket-name.eu-central-1.s3.amazonaws.com:443;

time="2021-05-11T21:34:15Z" level=info msg="Beginning to serve on port:9655"
time="2021-05-11T21:34:15Z" level=info msg="s3 name 'eu-central-1' available at s3 endpoint 'https://bucket-name.eu-central-1.s3.amazonaws.com:443' will be monitored"
time="2021-05-11T21:34:15Z" level=info msg="listenPort ::9655"
Connection to S3 endpoint failed : RequestError: send request failed
caused by: Get https://bucket-name.eu-central-1.s3.amazonaws.com:443/: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not bucket-name.eu-central-1.s3.amazonaws.com
panic: inconsistent label cardinality: expected 1 label values but got 0 in []string(nil)

goroutine 10 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
        /go/src/github.com/prometheus/client_golang/prometheus/value.go:107
main.S3Collector.Collect(0xc000088540)
        /build/main.go:84 +0xd3b
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
        /go/src/github.com/prometheus/client_golang/prometheus/registry.go:446 +0x19d
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
        /go/src/github.com/prometheus/client_golang/prometheus/registry.go:457 +0x571