Comments (13)
Do you get additional log messages in the startup_*
files that capture stdout & stderr? The temporary hidden node won't forward its logs messages to the parent node's logger. I'm wondering if they end up somewhere.
from rabbitmq-server.
One thing to note is what the classic config peer discovery plugin returns when asked for the discovered nodes: it returns rmq2
, rmq1
and the local node (rmq0
, rmq1
or rmq2
). This means rmq1
and rmq2
know about themselves twice, but never hear of rmq0
.
I'm not sure this can explain the problem, but I'm curious to know why that is.
from rabbitmq-server.
Difficult to say. I don't know if a peer discovery backend could return duplicate without it to be a wrong value.
About the classic config one, we could log several a warning:
- if there is a duplicate
- if the current node is missing from the list
We could also return a fatal error if the current node is not in the list.
I don't know enough how they are used in production unfortunately.
from rabbitmq-server.
This is strange, because I do remember testing this with 3.13 when the peer discovery changes were being made.
Next step will be to reproduce this using the generic-unix package so @dumbbell can look into it.
This could just be my test environment 🤷♂️
from rabbitmq-server.
I will check that out in my docker environment.
I did post a follow-up comment the other day that I must not have saved. I created the following repo which will bring up a 3-node cluster using inter-node-TLS and rabbitmq/rabbitmq-server@main
:
https://github.com/lukebakken/rabbitmq-server-11534
It worked fine the other day, but I will return to it today to make sure, as well as test with the 3.13.x and 3.12.x branches.
This issue may be just my docker project, or something else in my environment. Stay tuned.
from rabbitmq-server.
Running everything on localhost
using rabbitmq/rabbitmq-server
main
and v3.13.x
branches work fine, but using docker and 3.13.3
images just doesn't work. If I use 3.12.14
images my cluster starts fine.
@dumbbell here are logs from starting up via docker compose:
make-up.txt
rmq0.log
rmq1.log
rmq2.log
from rabbitmq-server.
AHA! @dumbbell I think these are the log files you're looking for:
[email protected]
[email protected]
[email protected]
Looks like this is the situation:
2024-06-25 22:15:41.286649+00:00 [info] <0.276.0> Peer discovery: node '[email protected]' selected for auto-clustering but its DB layer is not ready; waiting before retrying...
2024-06-25 22:15:42.622963+00:00 [info] <0.276.0> Peer discovery: node '[email protected]' selected for auto-clustering but its DB layer is not ready; waiting before retrying...
...investigating...
from rabbitmq-server.
If I use the "regular", non TLS-encrypted distributed Erlang version of my project, it works fine for clustering with 3.13.3 (https://github.com/lukebakken/docker-rabbitmq-cluster/tree/main). So, this issue must have something to do with the hidden peer node and proto_dist inet_tls
🤔
from rabbitmq-server.
Logs from starting lukebakken/docker-rabbitmq-cluster
using this docker image:
rmq0.log
rmq1.log
rmq2.log
from rabbitmq-server.
Aha, I don't think I would have noticed that!
> rg -A 1 'backend returned the following config'
rmq2.log
2732:rmq2-1 | 2024-06-26 18:22:45.313021+00:00 [debug] <0.276.0> Peer discovery: backend returned the following configuration:
2733-rmq2-1 | 2024-06-26 18:22:45.313021+00:00 [debug] <0.276.0> {ok,{['[email protected]','[email protected]','[email protected]'],disc}}
rmq1.log
2390:rmq1-1 | 2024-06-26 18:20:43.817877+00:00 [debug] <0.276.0> Peer discovery: backend returned the following configuration:
2391-rmq1-1 | 2024-06-26 18:20:43.817877+00:00 [debug] <0.276.0> {ok,{['[email protected]','[email protected]','[email protected]'],disc}}
rmq0.log
2390:rmq0-1 | 2024-06-26 18:20:43.746033+00:00 [debug] <0.276.0> Peer discovery: backend returned the following configuration:
2391-rmq0-1 | 2024-06-26 18:20:43.746033+00:00 [debug] <0.276.0> {ok,{['[email protected]','[email protected]','[email protected]'],disc}}
...that's definitely an issue.
from rabbitmq-server.
@dumbbell I owe you 🍻 the next time we meet up:
I don't think there are enough facepalms for this.
from rabbitmq-server.
Event if there is a configuration error, RabbitMQ shouldn't leave you alone. You had no easy way to spot the problem. RabbitMQ should give some warnings and not block like this.
from rabbitmq-server.
At least now I know what to look for in the debug output. Should we log a warning if a peer discovery backend returns duplicate identical node names?
from rabbitmq-server.
Related Issues (20)
- Failure to export some runtime parameters in rabbitmqctl export_definitions
- In production, nightly restarts of two nodes have led to frequent errors such as "operation queue.declare caused a channel exception not_found". HOT 1
- Try to improve error message when OpenSSL/FIPS can't access configuration file.
- Pass MQTT username and client_id to AMQP headers HOT 1
- Add message size metrics export to Prometheus
- Crash on RPC reply on 3.13.4 while it works on 3.12.14 HOT 2
- How to config keycloak for RabbitMQ
- Provide detailed memory metrics via prometheus plugin HOT 6
- The file descriptor metric reported by Prometheus is inaccurate
- ERROR: epmd error for host rabbitmq-test-server-0.rabbitmq-test-nodes.rabbitmq-test: timeout (timed out)
- `pre_init failed in system quorum_queues` when upgrading from 3.13.4 to 3.13.5 HOT 16
- AMQP 1.0: should default max frame size be/reported as `unlimited`? HOT 2
- Crash in credit reply on quorum queue leader change
- MQTT with OAuth 2: investigate if credential expiration can be checked before or after a keepalive frame is sent HOT 2
- HTTP API endpoint that returns queue metrics runs into an exception HOT 5
- Cancel AMQP stream consumer when local stream member is deleted
- AMQP 1.0: log client-provided connection name (when available) after accepting a connection HOT 2
- `mqtt_unexpected_call` error when closing MQTT connection
- Streams replica can fail to recover and fail to accept consumers in case of an abrupt (`kill -9`-like) node process termination on Windows HOT 4
- Shovel HTTP API: GET /api/shovels/{vhost}/{name} incorrectly returns a list of shovels
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-server.