Improve / abstract serialization and deserialization about pulsar-rs

[Geal]

there might be a simple way to do that: remove serialization and deserialization from the producer and consumer API, and instead provide a nice API around the messages. Instead of having send, send_json, send_raw, let's just have send_message, but provide code examples for serialization.
On the consumer side, let's have a Stream of messages that will be parsed directly by the client code.

It will make simplify integration with other formats like Avro or Protobuf, and using the schema registry. It will also make message batching much easier to implement.
what do you think?

[stearnsc]

We have the SerializeMessage and DeserializeMessage traits; how would you feel about having send_message be generic over T: SerializeMessage or similar? (and same for the Stream impl on the consumer). Or are there things that just working with &[u8] and Vec<u8> buys us from an implementation perspective that I'm missing?

[stearnsc]

I went through and pulled the serde dependencies in favor of the above traits, and also made some other minor restructures (send_raw taking a Message instead of bytes, etc). It's on #45 if you want to take a look before I merge it in (and obviously I haven't published it yet, so time to change stuff if you foresee issues).

[Geal]

I'll look at it later today, thanks

[Geal]

from what I see, it can be useful to have some state management in the serialize/deserialize part (counting, message indexes, chunking, etc). Which is hard with the traits as they are. And if we need to get a schema dynamically (which is a thing apache people like to do) depending on the message, it will be hard to do from inside the trait.

[stearnsc]

As I see it, there's nothing stopping users from making the DeserializeMessage a pass-through and handling deserialization manually while consuming the Stream (allowing then to do whatever they want) - do you think that's a bad API for e.g. the apache use case? If not, I should add a DeserializeMessage impl for consumer::Message so users wouldn't need to newtype.

[Geal]

that should be fine for now, you're right

[Geal]

cf #92