koivunej / eventstore-tcp

Tokio-based EventStore client API in Rust

Post 2017 update

Long story short: Apologies if you came here looking for a client library. You should look into YoEight's eventstore-tcp.

I originally started creating this related to my school work. Since then, tokio-proto with it's multiplexing support went away and I haven't really touched this in a long while. I haven't ever used this as a client library for anything, but instead I have used this crate to implement the server side of eventstore but then only using the pb types and adaptors on a custom multiplexing layer.

---

Tokio-based EventStore client API written in Rust

EventStore is an open-source immutable event database. EventStore allows writing to named event streams with optional optimistic locking and reading from a stream or a stream of all events in the database (not yet implemented in this project). This project aims to be a driver with similar features to the .NET client API but at the moment it has the simplest operations implemented:

• read an event
• read a stream forwards or backwards
• read all events forwards or backwards
• write events
• delete stream

Supported server version: currently this has been developed against the latest stable 3.9.3.

The suffix -tcp refers to the fact that EventStore also has AtomPub-based HTTP API. There exists already a crate for that: http_event_store. The TCP protocol is multiplexed, custom framed with the payloads specified as protobuf messages. Some of the payloads are exposed by the current version of this API.

Examples and usage

Please see the documentation.

The repository also includes an aspiring command line client under testclient/.

This crate is not yet available on crates.io as it depends on a custom fork of tokio-proto but you can try it out by adding this to your Cargo.toml:

[dependencies]
eventstore-tcp = { git = "https://github.com/koivunej/eventstore-tcp.git" }

Unimplemented features

1. read events from $all stream
2. deleting a stream
3. adapted interface for DeleteStreamCompleted
4. volatile subscriptions:

• refactoring to use tokio_proto::streaming::multiplex instead of tokio_proto::multiplex
• current messages are ok as headers, but appeared events could probably be body chunks
• maintaining a subscription by pumping events to a futures::sink::Sink, detecting overflows and dropping the subscription

5. Less of directly using the protobuf messages in the API
6. Cleaning up the message builders
7. Hide the use of Package from users
8. Add some "operation" API so that user does not need to match package.message {}
9. Nice API which would not require users to run the `tokio_core::reactor::Core``

"Perhaps later" features

1. persistent subscriptions (haven't researched this yet)
2. competing consumers (?)
3. long running transactions (???)

Contributing

Currently the code base is not formatted with rustfmt as the code generated by quick-protobuf is very slow to process by rustfmt. Later on hopefully rustfmt can be used, hopefully with default options which hopefully somewhat matches the current style.

As a fork of tokio-proto is currently used you need to clone this repository with --recursive:

git clone --recursive https://github.com/koivunej/eventstore-tcp.git

Building

cargo build will handle building, and testclient becomes usable after building it in it's own directory: cd testclient && cargo run -- --help.

Testing

Currently cargo test runs very primitive codec tests to ensure everything pretty much works. Also, I'm not sure if there is any point in aiming at 100% coverage en/decoding protobuf messages. Perhaps later on if server version differences are discovered these will make more sense.

The testclient/ contains test_with_inmemory_es.bash which will do some smoke testing against an EventStore instance expected to be running at 127.0.0.1:1113. The script also includes code to fetch but not run an inmemory instance as it's currently impossible to download EventStore binaries from an https:// host (if you know one, please let me know) and either way this should probably be done with some jailing/sandboxing to be accessible.

Simple no-brainer way to run the server is to spin-up a local ubuntu VM based on something similar to a VagrantFile below (assuming you have vagrant and VirtualBox ready to go):

Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.network "forwarded_port", guest: 1113, host: 1113
  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
    vb.memory = "2048"
  end
end

Then download the latest EventStore tar.gz and execute it with ./run-node.sh --ext-ip 0.0.0.0 --mem-db.

Rebuilding the client_messages

1. Obtain ClientMessageDTOs.proto or raw link
2. Checkout latest quick-protobuf: git clone https://github.com/tafia/quick-protobuf
3. cd quick-protobuf/codegen
4. cargo run --single-mod --output $es_tcp_checkout/src/client_messages.rs ClientMessageDTOs.proto

Modify the generated file like:

-#[derive(Debug, Default, PartialEq, Clone)]
+#[derive(IntoOwned, Borrowed, Debug, Default, PartialEq, Clone)]

License

MIT.

eventstore-tcp's Issues

Persistent subscriptions

haven't researched this yet

Replace rustc-serialise by Serde

...even if this is a dev-dependency, rustc-serialise has become deprecated.

Volatile subscriptions

• refactoring to use tokio_proto::streaming::multiplex instead of tokio_proto::multiplex
• current messages are ok as headers, but appeared events could probably be body chunks
• maintaining a subscription by pumping events to a futures::sink::Sink, detecting overflows and dropping the subscription

Adapted interface for DeleteStreamCompleted

@koivunej please edit me 😄

Add some operation API so that user does not need to match package.message

@koivunej please edit me 😄

Support volatile subscriptions

Making this issue to document what I've learned on subscriptions share my current thoughts on how to possibly proceed.

As I mentioned in another thread, the sequence of messages seems to be:

C: request_id: 1, payload: "subscribe to Foo"
S: request_id: 1, payload: "subscription started"
...
S: request_id: 1, payload: "notification on Foo"
... until either
C: request_id: 1, payload: "stop subscription"
S: request_id: 1, payload: "stopped because you unsubscribed"
...or...
S: request_id: 1, payload: "stopped because $reason"

Using tokio_proto::streaming::multiplex would almost allow supporting the whole conversation with notifications and "stopped because $reason" being the body chunks. However client sending "stop subscription" would have to somehow bypass the multiplex layer altogether, which I do not think is possible.

Other way would be to write a custom, more general multiplex support that would effectively create separate (Sink, Stream) pair for each request_id. As noted in the linked tokio_proto issue discussion, there might be more usage for such implementation than just the ES protocol.

Other thing the custom "more general multiplex support" could support is server sending something which is not in response to anything client requested. With ES these would be the heartbeat messages sent by server.

CustomTryFrom/Into should be replaced by TryFrom/Into

It seems CustomTryFrom/Into types are a custom version of the official TryFrom/Into type but forcing the try_from/into methods to return Self in a tuple if an error occur.

I now that you make these Traits because you wont use the unstable ones. That's a really strong argument, Yep ! the eventstore-tcp library is in an early-pre-alpha version (0.0.1) for now, we can publish a beta version when all of the useful stuff is stable.

This makes the library less flexible with the standard operators and language inference.

As a little example, in this snippet I show you how easy it is to convert error types without using the direct closure syntax (|_| { ... }) and just the method you want to apply (From::form) instead. It seems dumb to remove the CustomTryForm/Into types just to use this feature but think about this little improvement for all of the recursive errors you return all around your code ? I need to know why you return Self according to the Self::Error.

As an old Rust user I can say this: "This is not Rusty!" 😄

No offense, I need to have your feedback and I will work on this library improvement (with you if you want/have time to). We need to make better experience for the developers too.

PS. I really believe that less is more, removing write-handed official-equivalent code seems to be the right way for me. And I have time to loose on this library (I ❤️ geteventstore).

Better examples

As the testclient is 99% of formatting and command line parsing and 1% of examples, it would be better to have simple "do one logical thing" examples under examples/. One logical thing could be:

• writing some hello world events to a stream
• reading stream events from beginning
• reading stream events backwards until a specific one is found

These reading ones would of course need to populate a stream, which would then demonstrate reusing of the client.

Long running transactions

@koivunej please edit me 😄

Less of directly using the protobuf messages in the API

@koivunej please edit me 😄

client: Responding to server-sent HeartbeatRequests

Server might occasionally send an raw::HeartbeatRequest which should be answered as raw::HeartbeatResponse with the same correlation id. Probably the easiest way to implement this is by creating a custom transport (AsyncRead + AsyncWrite) implementation that will handle this before the package gets to the multiplex stage like in example tokio-line/ping_pong.rs.

Competing consumers

See if Byteorder dependency could be dropped in favor of bytes

Currently codec uses byteorder crates extension traits to read and write little endian values. The BytesMut is apparently convertable into a Buf by just:

let mut buf = &mut bytes[..].into_buf();

Then migrate all read_u32 etc to using methods on bytes::Buf.

Hide the use of Package from users

@koivunej please edit me 😄

testclient: better $all handling

Currently testclient fails at least the following cases unexpectedly.

1. testclient read '$all' --position first fails with Read failed: Event(NoStream)
2. testclient read '$all' --position 0,0 hits an assertion

This is because ReadEvent command is sent in error. Perhaps the --count should default to something else when using $all, for example 10?

Nice API which would not require users to run the tokio Core type

@koivunej please edit me 😄

Vagrantfile for testing against inmemory EventStore

There is a simple shell script testclient/test_with_inmemory_es.bash which currently funtions as a smoke test mostly for testclient. It'd be great to get a Vagrantfile that'd:

1. install eventstore 3.9.x during provision (vagrant up)
2. set up port mappings so that it can be used for testing with test_with_inmemory_es.bash immediatedly
3. start it up, running in-memory (--memdb)

There is a minimal Vagrantfile example in the README in the testing section.

Eventually it'd be great if we could use the same script to manage ES during travis builds. Since the travis build already runs in a sandboxed environment there shouldn't be ES needed. It might be possible to use the debian packages provided by EventStore.

RawMessage and AdaptedMessage

We need to talk 😄

What is the real difference between the RawMessage and the AdaptedMessage ?
Why did we don't merge the two types in one unique with more options ?

There is too much conversion between them:
ProtobufType -> RawMessage -> AdaptedMessage -> RawMessage

This is too much to handle, too much errors and wheels. The classic ProtobufType -> EventStoreMessage can be a great improvement to this lib'.

I need cons' and pros' about this idea.

Cleaning up the message builders

@koivunej please edit me 😄