MySQL Binlog subscriber about watermill

[njern]

Do you guys have thoughts around tapping into the PostgreSQL equivalent (a WAL subscriber) ?

[m110]

I guess logical decoding could be used, as in debezium: https://debezium.io/docs/connectors/postgresql/
Although I'm not sure if there is a library for Go at the moment.

[joeblew99]

@roblaszczak
i would not use any of the Deebzium stuff. I dont like its approach.

for MySQL use this: https://github.com/uber/storagetapper

for PostreSQL use thunder: https://github.com/samsarahq/thunder/blob/v0.5.0/livesql/binlog.go

let me know if your curious

[roblaszczak]

@joeblew99 for now simple sql-queries based subscriber is available: https://github.com/ThreeDotsLabs/watermill-sql

We have in plans add also Binlog based subscriber, but we need more time and be sure that this simple implementation is stable :)

[joeblew99]

I had a look at https://github.com/ThreeDotsLabs/watermill-sql Its a bit obtuse as to what its doing exactly. It seems to be designed for polling the DB to see what changed ? - wow its brute force - it solves the issue of having the parse the query and match it to the topic based on the predicates which is a tricks problem.

…

On Tue, Aug 13, 2019 at 8:33 PM Robert Laszczak ***@***.***> wrote: @joeblew99 <https://github.com/joeblew99> for now simple sql-queries based subscriber is available: https://github.com/ThreeDotsLabs/watermill-sql We have in plans add also Binlog based subscriber, but we need more time and be sure that this simple implementation is stable :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5?email_source=notifications&email_token=AC3RU43HVFBBST2BOUXR7C3QEL5BHA5CNFSM4GE72FB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4GSJXI#issuecomment-520955101>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC3RU472HFQGCDEY4DASG43QEL5BHANCNFSM4GE72FBQ> .

[roblaszczak]

@joeblew99 so creating docs is in progress so it may be not clear, but in short:

• you are able to produce and consume messages from MySQL (but you can extend it for other SQL's if you need) tables
• it's using just SQL queries, it's slow but fast enough for a lot of projects (for projects where it's not - we will introduce binlog implementation
• and it's of course fully compatible with Watermill's Pub/Sub, so you can use router, middlewares etc.

[joeblew99]

@roblaszczak Thanks and this all makes sense.

[0michalsokolowski0]

I would like to work on it

[Jonathan34]

@0michalsokolowski0 wondering if you made any promising progress on that? i am probably fine with mysql select queries but just wondering