Comments (14)
@bmflynn @cressie176 @dvovney @samragu @artembilan
We merged #11339 yesterday into v3.13.x
branch.
This means 3.13.3 will revert to the 3.12 behaviour that the broker interprets the x-death
header from publishing AMQP 0.9.1 clients.
This fix will only be available for 3.13.
Starting in RabbitMQ 4.0, the broker will behave like 3.13.0 - 3.13.2, i.e. the broker will not interpret the x-death
header from publishing clients. Hence, use a customer header in 4.0.
from rabbitmq-server.
@ansd Absolutely. I've made the change and our retries seem to be passing the same tests.
I think this issue is closable from my perspective. Thanks!
from rabbitmq-server.
specifically hints that RabbitMQ provides the header, not that applications should or even can use it.
We disagree. The spring documentation clearly and explicitly recommends the opposite; that you can use the x-death count attribute to conditionally decide to retry or abort after a number of attempts. See below...
The loop continue without end, which is fine for transient problems, but you may want to give up after some number of attempts. Fortunately, RabbitMQ provides the x-death header, which lets you determine how many cycles have occurred
It has never been the goal to make applications use that header
Does the RabbitMQ documentation state that applications should not rely on all or certain x-headers added to the message by the broker? My understanding is that the x-
simply indicates headers that are RabbitMQ extensions to the amqp specification, rather than being for RabbitMQ's internal use only. Why would the RabbitMQ documentation go to such lengths to document features users were not expected to rely on?
and no guarantees that such behavior will never change.
Just because there are no guarantees that the behaviour won't change, does not mean the behavior should be broken without due consideration or warning.
in the light of a discussion in spring-cloud/spring-cloud-stream#2939 this is does not seem to be as relevant either way but
Again we disagree. I would argue the issue is even more relevant. spring-cloud/spring-cloud-stream#2939 is still open and shows how disrupting this breaking change has been.
from rabbitmq-server.
did you enable the message_containers
feature flag?
from rabbitmq-server.
Not explicitly, but it does show as "Enabled" in the management interface.
from rabbitmq-server.
@ansd and @kjnilsson discussed this internally and have found where the regression was introduced.
from rabbitmq-server.
from rabbitmq-server.
@bmflynn could you possibly explain a bit how you are using this feature? Collaborative editing of x- headers is a bit odd in my view. x- headers "belong" to the broker and using it this way feels somewhat odd.
It is very unlikely we'd provide the same feature for other protocols such as AMQP (1.0)
from rabbitmq-server.
Dead-lettering and the x-death['count']
value are used as part of a retry mechanism where a message with expiration is retried until x-death['count']
is greater than the maximum number of retries.
Essentially, when message handling fails the message properties (including headers) and data are published with a message expiration via an error exchange to a queue with the dead-letter exchange set to the original exchange. The message will remain in this queue to expire at which point it gets routed back to the original exchange and is retried. This mechanism continues until the original message is handled successfully or x-death['count']
reaches the configured maximum number of retries.
There is no editing of the x-headers other than the fact that they are part of the original message properties that are used as part of a new message that is sent to an error exchange.
What is the specific feature you are referring to? Do you mean it's unlikely that a death count, or the number of times a message is dead-lettered, will be available in AMQP 1.0?
from rabbitmq-server.
@bmflynn Starting with "message containers" in 3.13, we would like to treat any x-
headers as belonging to the broker, meaning a client shouldn't publish messages with x-
headers. (If a client does publish with x-
headers, these headers won't be interpreted in any special way anymore as illustrated by your example).
You are right that this is a behavioural change compared to 3.12 and arguably a breaking change belonging to 4.0.
So, we could "fix" / revert to the 3.12 behaviour by having the broker parse incoming headers and therefore interpreting the x-death
header coming from a publisher.
However, we very likely would remove such special interpretation again in RabbitMQ 4.0 (which probably ships end of this year).
Therefore, my question is: Could your client set a custom non x-
header and just increment it itself when it re-publishes the message?
Here is an example based on yours:
Code example with custom header
import os
import time
from collections import namedtuple
from pprint import pprint
from pika import BasicProperties, BlockingConnection, URLParameters
Msg = namedtuple("Msg", ["method", "properties", "body"])
amqpurl = "amqp://localhost:5672"
exchange = "amq.topic"
start_queue = "start_queue"
dlx_exchange = "errors"
error_queue = "dead_letters"
routing_key = "test_retries"
data = "XXX"
conn = BlockingConnection(URLParameters(amqpurl))
ch = conn.channel()
ch.exchange_declare(dlx_exchange, "topic")
ch.queue_declare(error_queue)
ch.queue_bind(error_queue, dlx_exchange, "#")
ch.queue_declare(start_queue, arguments={"x-dead-letter-exchange": dlx_exchange})
ch.queue_bind(start_queue, exchange, routing_key)
print(f"first publish {exchange=} {routing_key=} {data=}")
ch.basic_publish(exchange, routing_key, data, properties=BasicProperties(expiration="500"))
# Wait for message to error out
print("waiting\n")
time.sleep(1)
first = Msg(*ch.basic_get(error_queue))
ch.basic_ack(first.method.delivery_tag)
print("Received:")
pprint(first.properties.headers)
print("Second publish")
Key = "my-retry-count"
# Increment the retry count if it exists
first.properties.headers[Key] = first.properties.headers.get(Key, 0) + 1
first.properties.expiration = "500"
ch.basic_publish(exchange, routing_key, data, properties=first.properties)
# Wait for message to expire again
print("waiting\n")
time.sleep(1)
second = Msg(*ch.basic_get(error_queue))
ch.basic_ack(second.method.delivery_tag)
print("Received:")
pprint(second.properties.headers)
print("Third publish")
first.properties.headers[Key] = first.properties.headers.get(Key, 0) + 1
first.properties.expiration = "500"
ch.basic_publish(exchange, routing_key, data, properties=first.properties)
# Wait for message to expire again
print("waiting\n")
time.sleep(1)
second = Msg(*ch.basic_get(error_queue))
ch.basic_ack(second.method.delivery_tag)
print("Received:")
pprint(second.properties.headers)
ch.close()
conn.close()
from rabbitmq-server.
I'd like to emphasize that the official Spring Framework documentation regarding the RabbitMQ binder suggests utilizing the x-death header as the primary concept for implementing retry mechanisms in services.
The potential impact of this might come as a significant surprise to numerous developers once their existing retry implementations cease to function.
from rabbitmq-server.
@dvovney where exactly does the Spring Framework documentation suggests that a client should re-publish a message containing the x-death
header?
from rabbitmq-server.
@dvovney where exactly does the Spring Framework documentation suggests that a client should re-publish a message containing the
x-death
header?
I've provide a link,
The loop continue without end, which is fine for transient problems, but you may want to give up after some number of attempts. Fortunately, RabbitMQ provides the x-death header, which lets you determine how many cycles have occurred.
To acknowledge a message after giving up, throw an ImmediateAcknowledgeAmqpException.
Main idea is to rely on x-death header to count how many time message handling has been failed.
While throwing exception during message handling, message end up in DLQ and it's expected that rabbit increase this counter so we could count how many attempts there were already
from rabbitmq-server.
@dvovney in the light of a discussion in spring-cloud/spring-cloud-stream#2939 this is does not seem to be as relevant either way but
RabbitMQ provides the x-death header, which lets you determine how many cycles have occurred
specifically hints that RabbitMQ provides the header, not that applications should or even can use it. It has never been the goal to make applications use that header, and no guarantees that such behavior will never change.
AMQP 1.0 separates application metadata from "all other" metadata, possibly in fact for such reasons.
from rabbitmq-server.
Related Issues (20)
- Include x-death header in Stream messages HOT 1
- 4.x: reduce default maximum message size further (e.g. to 64 or 50 MiB) HOT 13
- Publish `amqp_client` 3.13.x GA versions to hex
- Emit event if configured queue length is reached
- Don't close connection if channel without finished publish is closed HOT 3
- [Prometheus] Reduce number of series for per-exchange/per-queue metrics HOT 8
- Logout action does not fully work on management UI
- Consul peer discovery: nodes can leave behind a service record in case of an unresolvable address HOT 2
- Output "last write" time for Streams
- Peer Discovery with DNS record wont cluster HOT 2
- 3.13.2: "x-opt-rabbitmq-received-time" only available for new messages HOT 5
- Allow Khepri post-migration step to not delete certain files or dirs HOT 1
- 3.12.x: AMQP 1.0 connection authentication fails with an external HTTP auth service HOT 11
- k8s peer discovery : rabbitmq pods are booting as a standalone nodes on ipv6 environment. HOT 4
- prometheus.filter_aggregated_queue_metrics_pattern has no effect HOT 1
- Shovels cant be deleted if they dont connect HOT 2
- rabbit_heartbeat.erl compile error with erlang OTP 27 HOT 2
- x-death header count is no longer incremented from RabbitMQ version 3.13.0+ HOT 14
- Direct reply-to during a rolling upgrade v3.12.14 -> v3.13.3: `function_clause` error in `mc_compat` HOT 1
- Bump up oidc-client library in rabbitmq-management
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-server.