Comments (21)
oh, I'm poking them as well ;) I also wanted to clarify why I'm not letting this go for this project as well.
from fastly-exporter.
by explicitly setting the __params_target label per target (and ideally using the host header and port from the request to set the target:port) we can get service discovery to work without
How should the fastly-exporter change?
from fastly-exporter.
I used https://github.com/pagarme/static-response-server to host the following . . .
It should not be necessary to run a separate process in order to use the fastly-exporter with http_sd_config.
from fastly-exporter.
I used https://github.com/pagarme/static-response-server to host the following . . .
It should not be necessary to run a separate process in order to use the fastly-exporter with http_sd_config.
Oh, that was literally just to test that the new http_sd endpoint json worked the way I wanted without having to diddle the golang code first.
I had already gotten your relabel_configs
variant working first before looking to see how we could iterate it.
from fastly-exporter.
Oh! OK, sorry for presuming :)
from fastly-exporter.
So instead of /sd
returning
[
{
"targets": [
"abc",
"def"
]
}
]
you want
[
{
"targets": [
"localhost:8080"
],
"labels": {
"__param_target": "abc"
}
},
{
"targets": [
"localhost:8080"
],
"labels": {
"__param_target": "def"
}
}
]
Is that right? If it works, this avoids some configuration in the prometheus.yaml — does it do anything beyond that?
Unfortunately, I'm not sure it can actually work — I don't think the exporter can serve a target host:port which is reliably correct. Consider the case of load balancers, or NAT, or anything which puts a middlebox between the client and the exporter. The host header isn't a reliable source of truth. I've fought this battle many times actually!! As far as I can see it's a properly hard problem for a process to know it's own address in a way that permits it to serve that address back callers reliably.
from fastly-exporter.
/cc @SuperQ if you have any input
from fastly-exporter.
ok, I was thinking we might have the problems you mentioned with the host header, so...
use the ip:port that is being connected to, as seen by the fastly-exporter, as the target?
that addresses at least some of the cases and for more complex ones folks can still do relabel_configs
to fix up if needed.
not great, but possibly better? we REALLY need a prometheus placeholder for http_sd that means "come back to the hostname you asked"
targets": [ "__meta_http_sd_url_host" ],
Which obviously CAN be done with relabel_configs
but I'd love to have it available implicitly so things work out of the box.
from fastly-exporter.
It would be lovely if it were possible! But I'm not aware of a way for a server to reliably deduce the address that a client used to establish an incoming connection. NAT, VPNs, proxies, etc. can all sit in front of the connection in a way that's invisible to the server. Middleboxes can intercept and rewrite traffic at the IP layer. DNS resolution happens before the initial handshake. And so on.
from fastly-exporter.
Unfortunately not feasible.
from fastly-exporter.
Well, wait a sec. you're right that there are a myriad of problems detecting from the server side, but in this case we would just need to provide a hint to the client, who does have enough information to make the right decision.
ala, "hey client, connect to me like last time"
So we just need to convince the http_sd folks to implicitly handle the hint on the prometheus side... ;)
from fastly-exporter.
As an interim step on our side, we can do as you mocked up as that at least reduces the complexity of the relabel_configs
while we negotiate with the http_sd
folks
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: service
- target_label: __address__
replacement: 127.0.0.1:8080
to
relabel_configs:
- target_label: __address__
replacement: 127.0.0.1:8080
"service": "<service>"
can optionally be added to the /sd labels
as well if we still want that explicit label
from fastly-exporter.
We can't do what I mocked up. "localhost:8080"
is not (necessarily) a valid target.
from fastly-exporter.
it should be, no? assuming just about every system under the sun has a localhost -> 127.0.0.1 lookup.
potential alternate suggestions:
- 127.0.0.1:8080
- fastly-exporter:8080
as really the target field (for fastly-exporter) is currently more used as a placeholder text string for replacement then an actual host/ip for binding
i.e. in our intermediate implementation, we don't need the target to be resolvable, we just need it to be consistent to be replaceable with relabel_configs
;)
from fastly-exporter.
...aaaand I have a snippet from my first attempt to not have to update the address in two places (http_sd_configs -> url
and __address__
)
- source_labels: [__meta_url]
regex: '.*://(.+)/.*'
target_label: __address__
replacement: ${1}
slightly wordier, but keeps up to date with whatever host:port is used in http_sd_configs -> url
via __meta_url
so folks don't have to update two places.
That actually makes it irrelevant what the target is actually called in the /sd json ;)
from fastly-exporter.
What about that? #85
from fastly-exporter.
...aaaand I have a snippet . . .
Does this actually scrape each service correctly?
from fastly-exporter.
Upon reflection, I don't think it's the job of the exporter to try to golf it's SD implementation to minimize the required configuration on the Prometheus side. I think we just do what Prometheus says we should do and leave it at that.
from fastly-exporter.
I'm not a fan of golf either (code or otherwise. possibly miniature?)
So, let me take the 10,000 foot view and explain what I'm beating this gong so hard
automated discovery of service discovery endpoints
About a year from now, folks are going to be like
Hey, http_sd is nifty (it is!) I'd like to autodiscover service discovery endpoints (not targets, endpoints) like what is available with /metrics endpoints in kubernetes for example.
ala
kubernetes
-> [autodiscover service discovery] <- hypothetical, but likely
-> [service discovery endoint(s) (e.x. fastly-exporter/sd)]
-> [target(s) (e.x. fastly-exporter-1/metrics)]
Now, /metrics autodiscovery is awesome as /metrics works out of the box. Any /metrics endpoint is processed the exact same way. ...However, currently fastly-exporter service discovery needs application specific relabel_configs
transforms to work.
Part of this onus is going to be on projects (like fastly-exporter) implementing http_sd
, but I think the other part is going to be on the dhttp_sd
docs and examples requiring service discovery endpoints work out of the box without relabel_configs
by default. Otherwise, if autodiscover is attempted with them, they will not work.
Because otherwise fastly-exporter and other implementers are going to have a gnarly time of it that we can more easily nip in the bud early.
This is why I think it is so absolutely important that fastly-exporter /sd work out of the box, not with less relabel_configs
(tho I did provide examples), but with none.
from fastly-exporter.
These are points to raise on the Prometheus repo, not this one, I think :)
from fastly-exporter.
Sorry, I didn't get a chance to review this quickly.
I don't think the exporter can serve a target host:port which is reliably correct.
Yes, this is going to be the tough part. For example, in our configuration we're going to have several exporters running different shards, with semi-dynamic hostnames generated by Kubernetes.
I haven't actually gotten a chance to test the existing implementation yet, as I am working on adding additional scrape target injection into our Kubernetes magement. I think that should be done soon, so I'll hopefully be able to report back before the end of the year.
from fastly-exporter.
Related Issues (20)
- Cannot pull new Docker image HOT 3
- Support HTTP3 metrics
- Better logging for rt.fastly.com (Client.Timeout exceeded while awaiting headers) HOT 2
- remove tls_version="any" from fastly_rt_tls_total HOT 3
- metric consolidation for http2/http3
- Race condition between processing and scraping
- carnality explosion with fastly_rt_datacenter_info HOT 3
- Consider adding a way to (optionally) track tokens HOT 2
- Reduce output size of metrics endpoint HOT 3
- `-metric-blocklist` doesn't work with `fastly_rt_datacenter_info` HOT 3
- Add all those wild new Compute@Edge metrics
- Missing Docker images HOT 9
- RT API meta-metric HOT 3
- Metric Timestamps HOT 15
- Add datcenters API enricment HOT 2
- Add partial docker tags HOT 4
- Continue starting up if fastly is down HOT 1
- Exporter should set it's own custom user-agent HOT 5
- Pagination is not working properly (not fetching every pages) HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fastly-exporter.