Comments (5)
Actually, the example I was using was from PyPI:
await producer.publish('mystream', f'msg: {i}'.encode())
It works if I replace above line with
msg = AMQPMessage(body='hello: {}'.format(i))
await producer.publish('mystream', msg)
Looking at the changelog, the example was modified with message change the example to AMQP 1.0
.
IMHO, it would be great to keep the first version of API as it is simpler.
Selection of protocol version probably shall be done differently, i.e. when creating producer.
from rstream.
I have looked into the code a bit. We can send
- raw bytes (is that AMQP 0-9-1?)
- AMQP objects (this seems to be AMQP 1-0)
- mix of above (publish batch allows that)
Is mixing of messages by design? This does not seem to be supported by consumer?
from rstream.
raw bytes (is that AMQP 0-9-1?)
No, you can store what you want. In general we decided to use amqp 1.0 to save the messages in this way all the clients are compatible.
But if you want you could use something else to save the messages.
Currently, all the stream clients support amqp 1.0 as default
from rstream.
Sorry for long delay. @wrobell, thanks for reporting this!
The errors you're getting are most likely because you published raw (non AMQP-encoded) messages to the stream and then tried to consume them with decoder=amqp_decoder
.
Indeed, mixing of differently encoded messages breaks consumer.
On the other hand, restricting publisher to use a single protocol doesn't seem good enough solution to me. Firstly, it doesn't really guarantee any data consistency in the stream - other publishers/applications can still write anything to the stream. So consumers will still have to deal with invalid messages.
Also, producer doesn't really "support" these protocols, it just treats anything that can be converted to bytes (basically anything with a __bytes__
method) as a valid message. I want to keep this feature since it makes extending library and using other encodings really easy.
I also think it would be helpful to wrap this error and make it more descriptive
from rstream.
@qweeze @Gsantomaggio Thanks for the information. I studied the code and the documentation bit more and it all makes sense now.
I wonder if the API could be improved, though. For example if AMQP is the default, then we should not have to specify the decoder when receiving a message. But then how to design the publisher method, so it uses AMQP by default and keeps message format flexibility?
BTW. While trying to understand how things work, I wrote some scripts to publish data to RabbitMQ streams on my own. My concerns are reconnecting clients (i.e. on broker restart) and performance (it can be improved by using custom asyncio protocol). Now, also above API dilemma 🤔 I will pour my findings into this project https://gitlab.com/wrobell/rbfly. Please do not hesitate to steal the ideas from it.
from rstream.
Related Issues (20)
- Create a method on producer and consumer to reconnect to a stream
- Provide a name to client connections
- Is rstream suppout External credentials HOT 1
- superstream: Close/Remove locator connection when not needed anymore
- avoid to keep locator connections opened
- NotImplementedError in basic_consumers /basic_consumer.py HOT 7
- OverflowError: int too big to convert HOT 3
- Implement Create/Delete SuperStream feature
- Is there something like deadletter queue in RabbitMQ Stream? HOT 2
- Future must be safely set_result HOT 3
- pypi: username/password authentication deprecated HOT 4
- review/refactor examples
- Preventing Excessive Callback Invocations After Consumer Closure in `consumer.subscribe` HOT 2
- Request for Callback Disconnection Feature After Receiving the Last Message HOT 1
- Offset specifications ignored when using SuperStream with stream plugin HOT 3
- Reviewing/Improving logging and exceptions
- Implement maxProducersByConnection and maxConsumerByConnection
- Fail to build on Mac M* series HOT 5
- Stream connection error while using rabbitmq and docker on remote server HOT 3
- Implement a builder or validator for the arguments
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rstream.