Comments (12)
Publishers are blocked on per-connection basis. Since aliveness test cannot use your apps connection, it cannot know that they are blocked.
A new monitoring endpoint may be worth adding but aliveness-test cannot provide information for your connections,
only overall node health.
from rabbitmq-server.
"when I put my queue into a state where it will not allow publishing new items"
this is not a correct statement. Connections can be in that (blocked) state but not queues.
from rabbitmq-server.
There are already HTTP API endpoints for connection info. If you'd like something to be added there, please start a discussion on rabbitmq-discuss.
from rabbitmq-server.
Thanks for the clarification, got the point it is of course not my apps connection and it can't know the app is blocked for publishing.
But this entry in the log:
Publishers will be blocked until this alarm clear.
Tells me as a user, that no publisher should be able to publish anything onto this RabbitMQ host, doesn't it? So the aliveness-test should not be able to do that as well?
Anyway - as a user tagged for monitoring, is there any way I can get the information, that RabbitMQ is in the alarm state where publishers will possibly be blocked?
from rabbitmq-server.
Aliveness test is part of a RabbitMQ plugin which can do/use things AMQP 0-9-1 clients cannot:
- There is a so-called direct Erlang client that uses distributed Erlang facilities instead of AMQP connections.
- Plugins can bypass many checks that
basic.publish
handler normally goes through.
and so on. Aliveness test is mean to test key broker infrastructure and not every possible failure scenario.
I need to take a look if management API can provide any information about alarms. Will get back to you.
from rabbitmq-server.
As a user I agree with @LarsFronius that the aliveness check should fail when publishing is blocked by the watermarks being exceeded. That the aliveness check passes when the queue cannot process messages is both surprising and disappointing.
from rabbitmq-server.
@CVTJNII only specific connections are blocked, so your claim that "the queue cannot process messages" is factually untrue even when alarms are in place. Like I said, we are VERY far from having a consensus about how the aliveness test should work. Everybody wants it to work the way their company's monitoring work. Sorry, we cannot accommodate all requests.
from rabbitmq-server.
@michaelklishin I respectfully disagree. Per Rabbit's output when the watermark is breached:
**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
Per the API documentation at http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_4/priv/www/api/index.html:
api/aliveness-test/vhost Declares a test queue, then publishes and consumes a message. Intended for use by monitoring tools.
So per the check's documentation it should publish and consume a message. However, per Rabbit's log in this scenario publishers are blocked. So I disagree that this is ambiguity in how the check should work, in my opinion the check is not operating as documented. That is the spirit of my objection, based on the documentation I've found I would not expect a 200 response when publishers are blocked.
Furthermore, when I hit this issue I had the watermark set to zero, and per the memory documentation at https://www.rabbitmq.com/memory.html I would expect all publishing to be stopped as per the log message above:
A value of 0 makes the memory alarm go off immediately and thus disables all publishing (this may be useful if you wish to disable publishing globally); use rabbitmqctl set_vm_memory_high_watermark 0.
If there is better documentation for the API please let me know, that was the best I found and aligned with other searches for how the endpoint should function.
So again, I disagree with the assertion that this is an issue with consensus on how the check should operate, and instead believe the check is not operating as documented. As the check is supposed to publish and then consume a message it should not pass when publishing is disabled.
from rabbitmq-server.
@CVTJNII publishers are blocked after they publish at least one message, which can (although not guaranteed and typically won't) be accepted fully, e.g. if it has a blank body. How would RabbitMQ know that a connection is publishing otherwise?
The aliveness check uses a regular RabbitMQ client and the only thing that could be missing is a timeout — which again, everybody has their own ideal default for.
So instead of assuming that the aliveness test should cover everything, please stop suggesting that we make it do A, B, C, …, X, Y, Z. Use multiple monitoring checks, the HTTP API provides a ton of data, and so does rabbitmqctl
, which now reports alarms in status.
from rabbitmq-server.
I'm locking this because this is getting ridiculous. I cannot think of a data service where a single request can discover every possible issue there might be (given that definitions of "healthy" varies from company to company, or even project to project). Yet somehow RabbitMQ is expected to provide it.
from rabbitmq-server.
Filed rabbitmq/rabbitmq-management#137 for one specific improvement we can make.
from rabbitmq-server.
@CVTJNII furthermore, you assume that when a publish method returns, the message is published. That's wrong: it simply means that the data was written to the socket. It absolutely does not meant that it has reached RabbitMQ, was read, parsed, and routed. When alarms are in place, connections that see a basic.publish
or content metadata or body frame will stop reading from the socket. However, the client knows nothing about that, even if it is actually blocked, unless it registers a connection.blocked
handler.
from rabbitmq-server.
Related Issues (20)
- A `rabbitmq-queues grow` equivalent is missing for streams HOT 2
- Channel Metrics Cardinality - /metrics stops working HOT 3
- Unhandled function clause exception when retrieving queue for nonexistent virtual host HOT 2
- Add the ability to specify `$node` in log formatter format string
- HTTP API: GET /api/queues/{vhost}/{name} can return duplicate keys for quorum queues HOT 3
- 3.11: two Raft replicas are in the timeout state, one is a candidate HOT 5
- OpenID compliance check is not based on the final specification
- Possible race condition in classic queue deletion/declaration handling for multi-node clusters
- Quorum queue replica Erlang process names can conflict between queues in different virtual hosts HOT 2
- OpenID Connect RP-Initiated Logout should be optional HOT 1
- Optimise how rdq files are scanned at CQ shared message store recovery startup HOT 21
- Management UI creates classic queue instead of virtual host default queue type in some circumstances HOT 2
- When I create a stream, an error message "could not connect osiris to replica" appears in the log. HOT 1
- Make it possible to configure OpenId Connect endpoints rather than discover them dynamically via OpenId Connect Discovery endpoint
- 3.13.0 - 3.13.2: Dead-letter cycle detection can wrongly drop messages
- 3.13.0 - 3.13.2: Wrong warning messages that dead letter messages get dropped HOT 1
- Streams: consider allowing non-numerical values for publishingId HOT 3
- Log tls handshake timeouts HOT 1
- Include x-death header in Stream messages HOT 1
- 4.x: reduce default maximum message size further (e.g. to 64 or 50 MiB) HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-server.