Comments (22)
Depending on the decision on prometheus/docs#543, it might be alright to not tolerate trailing whitespace.
from common.
Most people who run into this issue will probably realize after a while that they made a mistake. Very few will come here and complain. It is still surprising that no trailing whitespaces are accepted, which is made worse by the confusing error message. I would expect many brain cycles of humanity wasted on this.
If the Prometheus server rejected trailing whitespace, then fixing this issue would indeed also require changing Prometheus because I do think we should be consistent within the project, including the degree of tolerance our parsers apply. I'd see it as an argument to change neither the Prometheus server's behavior nor the text parser at this point.
But what if the Prometheus server accepted trailing whitespace? :philosoraptor:
from common.
Un-assigning from myself as I intend to focus on other Prometheus project. Right now, others are way more involved in text format affairs than I am.
from common.
I just ran into this. The error message with a trailing space is really confusing. It says text format parsing error in line 1: expected integer as timestamp, got \"\"
. But when you add a timestamp it says that's wrong, too.
from common.
I don't think we'll be making any changes here, if you use a client library it'll handle it for you plus OpenMetrics will supersede all this.
from common.
I guess a typical scenario where to run into this is the textfile collector of the node exporter. That's where all kind of whitespace usage will be encountered because the textfile snippets are often created in a makeshift way. From that perspective, I see value in allowing trailing whitespace.
A different story is if and when the textfile collector will switch to OpenMetrics. If it does, we'd impose the strict whitespace requirement of OpenMetrics to all the creators of textfile snippets. (In principle, I'd love to let the textfile collector converge on OpenMetrics, too, but I'd also prefer OpenMetrics to be liberal with whitespace, which isn't going to happen.)
from common.
My thoughts are that changing the Prometheus text format this late in the game is unwise, anything that it currently does is now defacto the standard. Keep in mind that there are parsers other than the ones we maintain, and this would be a breaking change.
from common.
We wouldn't change the creation code, we would only make the parser more tolerant. It doesn't have to be part of the standard, it's just following the https://en.wikipedia.org/wiki/Robustness_principle .
Obviously, it would be bad if we saw the parser as a test to certify standard compliance of Prometheus text format generators. I wouldn't see it that way, simply because the Prometheus text format is not a publicly advertised standard. (That would be OpenMetrics, for which those tests exist or will exist.)
from common.
Obviously, it would be bad if we saw the parser as a test to certify standard compliance of Prometheus text format generators
This code is the official parser. If it parses it, it's in compliance.
We wouldn't change the creation code, we would only make the parser more tolerant.
Thus that would be changing the standard.
it's just following the https://en.wikipedia.org/wiki/Robustness_principle
This turns out to be a bad idea, due to things like this.
I wouldn't see it that way, simply because the Prometheus text format is not a publicly advertised standard.
I don't see how that's relevant. We've always defined this code to be the reference implementation.
from common.
I don't follow that line of argument. It all hinges on the premise that the parser is a compliance test and that the parser code defines the standard.
For better or worse, that was never standardized or advertized.
We are in a gray area here. I would like to navigate this ambiguity to the benefit of our users. In this case, allowing trailing whitespace is what most people will expect (as proved by multiple people that have run into this). We'll help all of them.
The counter case to bring up here is that somebody might have implemented a parser that doesn't tolerate trailing whitespace, too. If now somebody creates text format with trailing whitespace that then somehow makes it to that 3rd party parser, there will be a problem. That's a lot of If's:
- A 3rd party parser has been implemented (reasonably likely), and it also rejects trailing whitespace (which is less likely – even our own intention was to accept trailing whitespace, it came as a surprise that we didn't, note that this issue is filed as a bug almost four years ago).
- Then somebody creates Prometheus text format with trailing whitespaces (likely, that's the whole point of fixing this issue), which is then fed to a parser that accepts trailing whitespace, and then the same text snippet is fed to a parser that doesn't accept trailing whitespace (which should be extremely rare, it's not that there are many opportunities to pass around makeshift constructed Prometheus text format – the most prominent example is the textfile collector, but those snippets are always ever read by the textfile collector).
from common.
For better or worse, that was never standardized or advertized.
It's been stated multiple times over the years. For example if promtool (which uses this) parses your /metrics then it is valid.
It has always been the reference implementation.
The counter case to bring up here is that somebody might have implemented a parser that doesn't tolerate trailing whitespace, too.
This describes both the current version of this parser and the parser inside Prometheus.
Differing versions of Prometheus, the node exporter or pgw doing different things for what is nominally all the v0.0.4 format is not desirable behaviour. We might have gotten away with this a few years back, but today changing this would require defining a v0.0.5 - which I don't think we're going to do.
as proved by multiple people that have run into this
Also as far as I'm aware we've only ever had one report related to this, for a user who has (presumably) long resolved their issue. It seems a bit disproportionate to introduce such risk/complexity for a use case that effectively doesn't exist. It's not practical to change the format every time a user comes up with a new form of invalid input.
from common.
from common.
The Prometheus server currently tolerates trailing whitespace.
from common.
That is a problem then.
from common.
Not accepting trailing whitespaces would not have been a big problem if the error message would not have said the exact opposite of what is actually the problem. That was the real problem.
from common.
The official parser and the Prometheus parser doing different things here is something that needs resolving, one way or the other. The Prometheus parser is wrong by the letter of the law, but this could be considered by some to be a breaking change.
Improving error messages is also good.
from common.
2.15 and master might differ on this.
from common.
Oh? Is this a recent regression?
from common.
mmh no. I was thinking that maybe the new parser was more strict about this. it is not.
from common.
The Prometheus parser is wrong by the letter of the law
This is again based on the premise that the expfmt
parser sets the standard. I already challenged that, so you could at least acknowledge that your understanding is not the common one.
IMHO, the letter of the law is https://prometheus.io/docs/instrumenting/exposition_formats/ . It says:
Within a line, tokens can be separated by any number of blanks and/or tabs (and must be separated by at least one if they would otherwise merge with the previous token). Leading and trailing whitespace is ignored.
That's crystal clear. The expfmt
parser has a bug, and the prometheus/prometheus parser is right.
from common.
Hello, just dropping in here to weigh in. I'm a newly minted prometheus user, though I'm have a fair bit of development experience.
Please, allow extra whitespace! 🙏 I'm glad the docs seem to come down on the right side of this.
The current behavior is confusing and discouraging. A format in this simple human-readable style sends a strong message that extra whitespace is ignored. At the moment, it's hard to figure it out that it's not.
I almost gave up on an exporter script because of this. I'm glad I pushed through. Not everyone will not have the time and energy to do so. This comment is the only reason I considered whitespace was my issue (thanks @beorn7!).
A friendly/helpful error message may be good low hanging fruit, but I'd love to see the underlying whitespace problem sorted. Hope this is useful, and let me know if I can help any other way.
from common.
Yes it's a bug because the docs say otherwise as implemented.
from common.
Related Issues (20)
- common v0.43.0 is not supporting kube-state-metrics v2.8.2 prometheus metrics format HOT 2
- NULL character on k8s zone labels
- Support environment variable for log level and format
- New release? HOT 3
- Implement changes needed to support UTF-8 metric and label names in Prometheus
- OAuth 2 client_secret should be optional
- LabelSet.String sorts incorrectly HOT 2
- Authorization credentials should be optional
- Update expfmt/text_parse.go to support the new UTF-8 syntax HOT 13
- SigV4 panics on GET requests / requests with nil body
- web: Support jwt/bearer token auth
- Breaking change in 0.48.0 HOT 10
- Version bump in prometheus/common breaks client_golang build HOT 3
- Prometheus failing to reload Probe TLS cert and key from disk HOT 2
- Prevent incompatibility issues between common and dependent repositories HOT 5
- OpenMetrics: _total suffix repeated multiple times HOT 7
- "error encoding and sending metric family" on HTTP/2 HEAD request
- Can client.key be removed as its giving security blocker in security scan? HOT 1
- Support for: INFO type
- Support for configuring ciphersuites in tls_config
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from common.