1.0 Release tracking issue about pulsar-rs

[Geal]

@stearnsc I think we can release the 1.0.0 from this commit: 7da2af2
it's stable enough (reconnection works well now) and the APIis a bit cleaner

[Geal]

@stearnsc FYI I'll get started on the futures 0.3 upgrade and will make it available in a PR. We're ramping up on the Pulsar deployment and I'll be in a good position to test the code before release

[stearnsc]

https://crates.io/crates/pulsar/1.0.0 🎊

[Geal]

thank you for this issue. From what I understood, you're already working on the futures 0.3 upgrade? Could you push the code in a branch somewhere so I can help?
On [1], since there are a lot of parameters, it might be good to have the constructor take a CommandProducer and CommandSubscribe as parameters and make builder interfaces for those (that's the approach we used in lapin: https://docs.rs/lapin/0.33.0/lapin/struct.Channel.html#method.queue_declare ).
On [2], it is especially tricky for in-flight messages: producers would have to buffer messages to resend them until the get a notification that the message was handled, and subscribers would need application specific behaviour for acknowledgements. Maybe this should live in a wrapper over the basic producer and subscriber API?

[Geal]

hey, I don't want to be pushy, especially now that we're in difficult times, but if you don't have the bandwidth for it right now, I can handle the futures 0.3 upgrade, starting from your current work or going from scratch. This is something that I'll need soon, and I'm happy to help

[stearnsc]

@Geal hey thanks so much for picking up the slack on this. We had some turbulence at Wyyerd, and the long and short of it is I haven't had the time to devote to our open source that I'd like, and the other dev here who worked a lot on open source is no longer with the company. We've hired more, and once everybody's up to speed I believe we will be able to lean back in, but that could be a couple months out. I've invited you to be a maintainer to hopefully help prevent this crate from dying in the meantime.

[Geal]

hey @stearnsc , thank you for the invitation, and I'm looking forward to your return :)

[Geal]

compression is done in #84

[Geal]

@stearnsc ping! I have advanced a bit, and I believe this crate should be ready for a 1.0.0-alpha release. Here's a summary of what I implemented in the last 2 months:

• upgrade async-await
• Protobuf schema update
• support for tokio and async-std
• connection through a pulsar URL (pulsar:// or pulsar+ssl://), with DNS lookup of the host, and TLS using the native-tls crate
• message batching in producer
• compression (in producer and consumer)
• automatic reconnection
• builder APIs for Producer, Pulsar and for messages (to set up properties and partition key)
• cleaned up a lot of internal code (removed the "engines" when possible to replace them with async functions or blocks if possible)

API changes (see 1c4ef59 for an overview):

• the executor is now a generic type instead of an argument (the current runtime can be obtained from a static method and both runtimes have been abstracted behind the Pulsar::Executor trait)
• Producer is now MultiTopicProducer, and TopicProducer is now Producer to match the consumer side
• Producer::send require a mutable producer (to allow reconnection)
• message deserializing is now done outside of the consumer (this simplifies iterating over a consumer stream)
• acknowledgements are now done through Consumer::ack (this simplifies the API, aligns behaviour with other clients, and helps in implementing reconnections)

About your points in the initial comments:

• Clean, stable consumer/producer API[1]: the current code heavily pushes towards builders, since I added more configuration options. It's getting quite nice to use
• Reconnection is not done yet in the best way possible (see #99, #100, #101) but it is already quite usable, and needed in projects like Vector (or our internal projects at Clever Cloud), so I'd prefer we ship the client as is and improves reconnection afterwards, since it will not break the API.
• producer batching is still a bit awkward because we wait for receipts (see #78)
• compression works quite nicely
• dead-letter topic is not implemented
• Finalized message serialization/deserialization API: I believe the current API is fine for now, if we want to make it more usable, we'll need to integrate support for json, avro, etc and managing schemas. This will be heavy to implement

Now I think we can get into testing and docs 😃

Issues that can be closed after this release: #28, #46, #49, #64, #68, #76, #80
New issues (not blocking the release): #78, #98, #99, #100, #101

[stearnsc]

This looks great! The issues around re-sending messages in the producer look tricky, but agreed that the current implementation seems good for the majority case.

Happy to release 1.0.0-alpha

[stearnsc]

@Geal is there a specific commit you think should be published as the alpha? It looks like HEAD has a few warnings, so I want to make sure I'm publishing the right thing (not opposed to publishing HEAD anyway since it's just an alpha).

[Geal]

81e38fa should be fine, I removed the last warnings. A clippy pass would be good before the final release

[Geal]

@stearnsc could you publish at 81e38fa for the 1.0.0-alpha?

[stearnsc]

Yep, sorry this has been done.

[Geal]

thanks :)

[stearnsc]

A few small things I think we should do before the 1.0 release:

• Heavily restrict the non-builder constructor APIs (maybe make all of them private?). We can always add them later, but if we include them, then we have to do a minor release for anything else we add (whereas adding to the builder pattern can be done non-breaking).
• Remove all the phantom-types from the builders. I like them in principle, but they're a messy part of the public API, which makes it more difficult to make non-breaking changes to the builders (even though it's unlikely to actually break any user code), and adds complexity to understand / maintain.
• Expose an escape hatch for the serialization API - right now there's no way to avoid copying data, so I think we should expose a send function on the producers that takes a producer::Message directly.
• Anything else?

If you agree, I can get these done today/tomorrow and release 1.0 tomorrow evening my time.

[stearnsc]

After having spent some time working on this, I think leaving the Topic as a phantom type in the builder is fine, and maybe the better solution is to just make defaults for the subscription (randomly generated?) and subscription type (Shared?) to simplify the interface. Mostly I just want to not have to change the number of type parameters on the builder, and I think we can live with the lifetime, the topic, and the executor.

[Geal]

Since there are more features now, the non builder constructors are a bit too big now so it's a good idea to hide them (especially seen from the breaking change angle).

I also agree with removing the types from the builders, they make the docs awkward, and safe defaults are fine. IIRC we could remove the lifetime since the Pulsar struct is clonable (and only contains Arc<_> members). The Topic type could be removed as well from the ProducerBuilder, since with_topic already transforms it into a String. For ConsumerBuilder we can keep it, the regex makes it explicit that we want a MultitopicConsumer

See also #109 to avoid copies when sending messages. There's probably no way to avoid copies entrely (if there's a serialization phase) but this way it leaves the decision to the user.

I wanted to have a user facing Message structure shared between the producer and consumer, to simplify the API and some patterns like copying from on topic to another, but they have very different behaviours

[stearnsc]

Playing around with it a bit, I'm wondering if it's worth hiding both consumers behind a unified public type. That would make it easy to automatically use a multi-topic consumer for a partitioned topic without the user having to know or care (we could expose access to such information for uses who care). If we went that route, it would be nice to do the same for the producer, but it's a little tricker because the interface is different.

[Geal]

In the Java client the consumer interface wraps both cases, and the builder can take one or more topics or a pattern: http://pulsar.apache.org/api/client/2.2.1/ On the producer's side, it's using one producer per topic. I'm not sure the multi topic producer is really needed, since creating and managing multiple producers is easy. While on the consumer side it makes sense: you'd often want to aggregate messages coming from multiple topics, and currently awaiting on multiple streams manually is a bit awkward.

…

On Sat, Jul 4, 2020, 14:29 Colin Stearns ***@***.***> wrote: Playing around with it a bit, I'm wondering if it's worth hiding both consumers behind a unified public type. That would make it easy to automatically use a multi-topic consumer for a partitioned topic without the user having to know or care (we could expose access to such information for uses who care). If we went that route, it would be nice to do the same for the producer, but it's a little tricker because the interface is different. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA5EAGI5XZETZFORWVBFZDRZ4OB7ANCNFSM4LGAIUSQ> .

[stearnsc]

Okay, I've rewritten the consumer to be an opaque type. This also allows us to do partition lookup on creation, so a partitioned topic automatically gets backed by a multi-topic consumer.

I did a similar partition-handling strategy for the producer (creating a producer to write to a partitioned topic will be backed by multiple producers that write round-robin), but I didn't attempt to unify with the multi-topic producer. I'm gonna leave the multi-topic producer in for now - we use it extensively here for our messaging pattern (large number of topics that get rarely written to), but I'm not opposed to moving it into an internal lib.

I still have some cleanup left to do, and want to do an audit of docs, but then I'll release this as alpha-2 so we can make sure the API changes aren't usage blocking, and plan to release 1.0.0 later this week.

[stearnsc]

Update on this - I've made a bunch of (hopefully) improvements, but in the process I've broken some of the tests. Planning on picking back up tomorrow.

[stearnsc]

I think we're in good shape here. I've published 1.0.0-alpha-2 so that those of us who use this heavily can integrate the new API and ensure there are no big blockers to the 1.0.0 release.

[Geal]

alright, that's great, I'll test it out

[stearnsc]

I've integrated into our systems and stuff seems to be working well, although I found and fixed a couple bugs (we were unwrapping instead of returning an error for a partially successful url parse, and when sending batched messages I was setting the message count to the batch size rather than the actual message count, so manually sending batches would fail). I've fixed both of those, and added a test for the batch for the batching, and published an alpha-3. My plan is to deploy the new version to our staging environment tomorrow and hopefully get some confidence there aren't any other lurking bugs, then if everything looks good, publish 1.0.0 over the weekend.

[Geal]

I'm updating our internal projects, I'll see before the end of the day if there are any issues. Could you take a look at #113 before the release though?

[stearnsc]

I've pushed a few more small tweaks, and think I'm ready for the release. I noticed you exposed the executor module - I'm not opposed, but I still don't understand the use case for it. Right now, a manual implementation of an executor is required to return an ExecutorKind, so my understanding is users are precluded from actually implementing their own executors, right? Or am I missing something?

[Keruspe]

The ExecutorKind should actually be ReactorKind, it’s only there because tokio has its own AsyncRead and AsyncWrite which are different from the ones from futures which async-std uses. It’s also used to « create » the TcpStream, using said reactor. It thus can be used to manually create any custom executor on top of the tokio or async-std reactor (let’s say some « distributed over network » proprietary executor for example), or a more verbose one such as the one from @Geal’s example

…

On Sat 11 Jul 2020 at 17:01, Colin Stearns ***@***.***> wrote: I've pushed a few more small tweaks, and think I'm ready for the release. I noticed you exposed the executor module - I'm not opposed, but I still don't understand the use case for it. Right now, a manual implementation of an executor is required to return an ExecutorKind, so my understanding is users are precluded from actually implementing their own executors, right? Or am I missing something? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABWNXQMYFNPTW3J2ZY7OPLR3B5DNANCNFSM4LGAIUSQ> .

[Geal]

There are use cases like the vector project where they have their own executor that carries state, debug info, statistics on tasks, etc, but delegates executing to either tokio or adync-std

…

On Sat, Jul 11, 2020, 17:01 Colin Stearns ***@***.***> wrote: I've pushed a few more small tweaks, and think I'm ready for the release. I noticed you exposed the executor module - I'm not opposed, but I still don't understand the use case for it. Right now, a manual implementation of an executor is required to return an ExecutorKind, so my understanding is users are precluded from actually implementing their own executors, right? Or am I missing something? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA5EAE2MQVAUQHBBPF4ERDR3B5DNANCNFSM4LGAIUSQ> .

[Geal]

Also the timers (interval, timeout) are specific to the reactor we use

…

On Sat, Jul 11, 2020, 17:32 Geoffroy Couprie ***@***.***> wrote: There are use cases like the vector project where they have their own executor that carries state, debug info, statistics on tasks, etc, but delegates executing to either tokio or adync-std On Sat, Jul 11, 2020, 17:01 Colin Stearns ***@***.***> wrote: > I've pushed a few more small tweaks, and think I'm ready for the release. > I noticed you exposed the executor module - I'm not opposed, but I still > don't understand the use case for it. Right now, a manual implementation of > an executor is required to return an ExecutorKind, so my understanding > is users are precluded from actually implementing their own executors, > right? Or am I missing something? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#67 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAA5EAE2MQVAUQHBBPF4ERDR3B5DNANCNFSM4LGAIUSQ> > . >

[stearnsc]

Ohh that makes sense. I was thinking the use was for new, non-tokio, non-async-std executors. @Geal unless you have any last reservations I'll go ahead and publish 1.0.0.

[Geal]

Fine by me 😀

[Geal]

🍾