Comments (14)
There is a set of changes related to dead lettering that have shipped in 3.13.2
or will ship in 3.13.3
: #10709,
#11173, #11160, #11159, #11174
from rabbitmq-server.
As for x-*
headers, nothing has been decided yet. There's a set of headers such as x-deaths
that were never meant to be modified by applications.
from rabbitmq-server.
@michaelklishin thank you for the link to #11174, it provides a lot more context, however I think it is still a bug and should not have been closed as a duplicate.
There's a set of headers such as x-deaths that were never meant to be modified by applications.
The application is not modifying these, other than when they move the message (by copying it) to a different queue.
Irrespective it's very confusing behaviour to have a "count" property set by the broker, which until recently was incremented with each rejection, but now if fixed at 1 no matter how many times the message is rejected. This was an undocumented change in v3.13.0, that breaks the current documented behaviour.
count: how many times this message was dead-lettered in this queue for this reason
from rabbitmq-server.
@cressie176 since 3.13.0
there were four changes directly related to dead lettering, including at least one related to the conditions under which the counter is incremented. They are all mentioned in release notes (including those pending release under release-notes
in this repository).
Use https://hub.docker.com/layers/pivotalrabbitmq/rabbitmq/v3.13.x/images/sha256-b32c328236bec5ec2b94bd4afd290687844a957919fe838457d10858c620bfc4?context=explore in order to compare.
If this is something distinct from all those behaviours, please put together an executable way to reproduce and we will consider reopening this issue or filing a new one.
from rabbitmq-server.
I searched the v3.13.0 release notes (the version which introduced the bug) prior to reporting it. They only mentioned dead lettering once and give no indication there was a breaking change relating to the way rejections are counted...
Messages are now internally stored using a new common heavily AMQP 1.0-influenced container format. This is a major step towards a protocol-agnostic core: a common format that encapsulates a sum of data types used by the protocols RabbitMQ supports, plus annotations for routing, dead-lettering state, and other purposes.
from rabbitmq-server.
since 3.13.0 there were four changes directly related to dead lettering, including at least one related to the conditions under which the counter is incremented
#10709 - This is the same bug, although it manifests with message expiry. The OP was able to workaround, but noted how this would likely cause issues for other users. @lukebakken acknowledges it is a regression and breaking change, but the issue was closed regardless.
#11173 - This issue is about adding the x-death header to streamed messages
#11160 - This issue is about a warning message when dead letter messages are dropped by the broker
#11159 - This issue is about dead letter messages being incorrectly dropped by the broker
#11174 - This discusses approaches for avoiding infinite dead letter cycles, and the introduction of deaths_v2 annotation
None of the above, other than the first which was closed without resolving the bug, are concerned with v3.13.0 of the rabbitmq broker failing to increment the x-death count field.
If this is something distinct from all those behaviours, please put together an executable way to reproduce and we will consider reopening this issue or filing a new one.
https://github.com/cressie176/rabbitmq-11331 demonstrates the bug
from rabbitmq-server.
@cressie176 apparently it was an intentionally decision (in the same discussion as #10709 (comment)) by several team members to keep the current behavior instead of reverting it only to change it back for 4.x.
Thank you for taking the time to put together an executable example to reproduce!
from rabbitmq-server.
The above decision was taken after investigating what would a message container-based 3.12.x era solution would be like #11043.
Time will tell how many paying users will ask for a temporary restoration of this behavior. Most seemingly can use a custom header when publishing, which is a fundamental solution which avoids modifying a header used internally by RabbitMQ to collect a series of dead lettering-related events.
from rabbitmq-server.
Thank you @cressie176 for the detailed issue and reproduction steps.
In https://github.com/cressie176/rabbitmq-11331/blob/7372dc0e7d69dc78dc52c27d8ad8f6b0f2889440/index.js#L17 you are publishing a new (i.e. 2nd) message. The broker won't interpret this x-death header from this new (2nd) message in v3.13.x.
The solution for you is one of the following:
- Use a custom count header as described in #10709 (comment), or
- Let RabbitMQ itself move the message back to the original queue, for example by configuring
x-message-ttl
on the dead letter queue.
I am unable to maintain this in a custom header since I would need to modify the message on rejection, which is not possible.
No, you don't need to modify the message on rejection, you can instead modify the message header when you publish your new (2nd) message.
from rabbitmq-server.
Thanks @michaelklishin and @ansd - It's good to know a fix was back ported to the v3 branch.
No, you don't need to modify the message on rejection, you can instead modify the message header when you publish your new (2nd) message.
This is what I was trying to avoid by using the x-death count attribute. Without a standard rejection count header, users won't be able to use the shovel plugin (or other 3rd party software) to move the messages from the dead letter queue back to the work queue in scenarios where they need to treat replayed messages differently from regular ones.
Rascal's scenario is that it needs to remove the immediate nack custom header, but given x-death count has been widely adopted by other frameworks to track rejections, there are certain to be other examples. Without the x-death count users will need to write their own shovel application. For Rascal, I've worked around this by using the x-death time value as well. Prior to v3.13.0 this was not updated by the broker. Ironically the same change that broke the count means a the timestamp is now set to that of the last rejection rather than the first.
For v4 I assume it won't be possible for the broker to keep a count of rejections of republished messages since there isn't a guaranteed message id. If not, can the x-death count attribute be removed so that it isn't misleading, and the removal listed as a breaking change in the release notes please?
from rabbitmq-server.
It's good to know a fix was back ported to the v3 branch
Do you mean the v3.13.x
branch? Which fix are you referring to?
Without a standard rejection count header, users won't be able to use the shovel plugin (or other 3rd party software) to move the messages from the dead letter queue back to the work queue in scenarios where they need to treat replayed messages differently from regular ones.
Shovels can be configured to set certain headers such as dest-publish-properties
, dest-add-forward-headers
, dest-add-timestamp-header
. Hence, users can treat messages replayed via the shovel plugin differently from other messages.
Ironically the same change that broke the count means a the timestamp is now set to that of the last rejection rather than the first.
Note that all these timestamps are wall clock time, and therefore an approximate point in time.
from rabbitmq-server.
Do you mean the v3.13.x branch? Which fix are you referring to?
From #11043
"This commit will only be packported to v3.13.x branch. In other words, 4.0 won't interpret x-headers as done in 3.13.0 and 3.13.1".
Have I misunderstood?
Shovels can be configured to set certain headers such as dest-publish-properties, dest-add-forward-headers, dest-add-timestamp-header
Thank you. I was not aware of these. The documentation says that dest-publish-properties is unsupported though, and there still doesn't appear to be the ability to increment the number of times the message was rejected.
Note that all these timestamps are wall clock time, and therefore an approximate point in time.
Thanks for the clarification. This will be sufficient for Rascal's use case.
from rabbitmq-server.
To clarify: #11339, which to some extent helps projects that publish messages with x-deaths
, is v3.13.x
-specific and will not go into main
for 4.0.
from rabbitmq-server.
Have I misunderstood?
Yes, #11043 is closed. Closed means not merged.
However, we merged #11339 yesterday which fixes the issue in 3.13.3.
from rabbitmq-server.
Related Issues (20)
- Consul peer discovery: nodes can leave behind a service record in case of an unresolvable address HOT 2
- Output "last write" time for Streams
- Peer Discovery with DNS record wont cluster HOT 2
- 3.13.2: "x-opt-rabbitmq-received-time" only available for new messages HOT 5
- Allow Khepri post-migration step to not delete certain files or dirs HOT 1
- 3.12.x: AMQP 1.0 connection authentication fails with an external HTTP auth service HOT 11
- k8s peer discovery : rabbitmq pods are booting as a standalone nodes on ipv6 environment. HOT 4
- prometheus.filter_aggregated_queue_metrics_pattern has no effect HOT 3
- Shovels cant be deleted if they dont connect HOT 2
- rabbit_heartbeat.erl compile error with erlang OTP 27 HOT 2
- Direct reply-to during a rolling upgrade v3.12.14 -> v3.13.3: `function_clause` error in `mc_compat` HOT 1
- Bump up oidc-client library in rabbitmq-management
- Load oauth javascript dependencies only when they are required
- Is it possible to configure consumer for multiple queues with priority? HOT 1
- list_unresponsive_queues crashes if a QoS 0 queue used by an MQTT client is present HOT 1
- The port range of the stream replication is modified
- rabbit_quorum_queue:all_replica_states doesn't handle some member states
- management SSL Cannot enable IPv4 and IPV6 HOT 1
- Peer discovery using inter-node TLS not working with 3.13.x HOT 13
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-server.