Take rust-bitcoin-rpc under rust-bitcoin org umbrella? about rust-bitcoin

[apoelstra]

I also like it.

@stevenroose what do you think about separating rust-jsonrpc into two crates -- a HTTP-agnostic one where we can require rust 1.15 (I think we need serde_json support so 1.14 is out ... I don't want to use strason because it has nontrivial unsafe code that I can't even check with MIRI because of open stdlib+miri issues, plus it seems to interact badly with serde 1.0 features like #[serde(untagged)], plus it's basically unmaintained, plus it's not very efficient).

The other crate would be rust-jsonrpc-hyper-0.10 or something like that, which would newtype the hyper::Client object to implement some rust-jsonrpc trait.

I think under those conditions we could reasonably pull both projects under rust-bitcoin, and also rust-bitcoin-rpc (assuming @jeandudey is ok with it). That crate is also probably where the rust-bitcoin Decimal and UDecimal types should live.

[stevenroose]

Here I support calls without macro acceptably readable: https://github.com/stevenroose/rust-bitcoindrpc/pull/1/

Here are the two method (notice the awkward handling of the arguments):
https://github.com/stevenroose/rust-bitcoindrpc/pull/1/files#diff-31bbf71c54bca98a0ae3d40a327af940R271
https://github.com/stevenroose/rust-bitcoindrpc/pull/1/files#diff-31bbf71c54bca98a0ae3d40a327af940R321

[dpc]

@laanwj @tyrel86 - seemed like you might be interested in it too.

[apoelstra]

I like this library and have a lot of respect for jeandudey. Unfortunately it depends on my own rust-jsonrpc, which we can't bring under rust-bitcoin because (a) it has a huge dep tree, as a consequence of depending on hyper 0.10 and (b) it can't compile under rust 1.14, also because of hyper 0.10. So I'm going to vote no on this.

If we could get to a point where rust-jsonrpc could reasonably be pulled under rust-bitcoin, I expect this library would also fit. But as near as I can tell, there are no HTTP clients out there that meet our requirements. (In addition to the problems I listed, hyper 0.10 is unmaintained...and the maintained version of hyper has a far worse dependency story.)

[dpc]

Makes sense. Thanks! I'll close, but if anyone has any comments, please go ahead.

[laanwj]

yes this would be nice, though as said the dependency on hyper is unfortunate; would be great to have a no-bells-and-whistles HTTP client which doesn't depend on anything besides basic network functionality

another project to consider maybe putting somewhere 'official' would be https://github.com/laanwj/rust-clightning-rpc

[stevenroose]

I've been working on a similar lib: https://github.com/stevenroose/rust-bitcoindrpc
TBH, I somehow assumed rust-bitcoin-rpc was incomplete and unmaintained. If that was not the case, we could perhaps join efforts..

Btw, @apoelstra how about changing jsonrpc to be generic and taking an "HTTP roundtripper" as an argument? That way the lib could fit, the bitcoindrpc lib would be equally generic and the user can just fulfill the roundtripper trait for their own HTTP client. There could also be third-party libs like rust-jsonrpc-hyper or rust-jsonrpc-reqwest, ... that implement the trait for popular HTTP clients. That way the burden of dependency management is on the users...

[dpc]

@stevenroose Argh.:D . I want to be mining on top of the longest chain, not on some fork!

All these libraries are pretty much about structs definitions, which typically can be just copy-pasted, so all hope is not lost. But I would still have us all converge on one "official" impl.

I looked briefly at your repo and one thing that I dislike in both @jeandudey and your code is macros. Macros are powerful, sometimes necessary, but generally should be avoided IMO. They make life harder for the tools, humans (especially newcomers) and they basically save just a couple of keystrokes which are one time-cost.

Anyway, maybe we can get a gitter channel or something, discuss what we want, why we do things like we do, argue for a while, and then pick one repo, move all the existing serialization types there, and then just keep working on that one version. Otherwise it's just silly to have bunch of incomplete self-made ones.

And maybe rust-bitcoin org could then mention it. Or maybe we could have rust-bitcoin-nursery org or something.

[apoelstra]

I'm going to reopen this because there's a wider discussion we should have about

• when it's reasonable to NIH to avoid dependencies
  • none of us have the bandwidth or expertise to implement a full HTTP client; on the other hand we don't need very much of HTTP to support a basic jsonrpc client
  • OTOH, we have had repeated minor-releases-breaking-our-code issues specifically with hyper and its dependencies
• when it's reasonable to require 1.14
  • for rust-bitcoin, which does no networking or I/O, this is mostly easy, just annoying (no custom #derive and no associated constants are the biggest pain points IMO)
  • but as soon as you want networking then you need a newer stdlib or net2 or something; if you want HTTP it seems you need tokio and friends, etc.; if you want unix sockets you need a new stdlib or unix_socket
  • also, for stuff like rust-lightning or rust-jsonrpc, they're probably only going to be used on full-featured applications that will require newer compilers and standard libraries. vs rust-bitcoin which is mainly data structures and can be used in lots of obscure places (or rust-secp which is mostly FFI and (nearl)y allocation-free)

BTW @dpc we have an IRC channel on Freenode -- #rust-bitcoin. I guess we should mention that in the README of all of our projects.

[laanwj]

Btw, @apoelstra how about changing jsonrpc to be generic and taking an "HTTP roundtripper" as an argument?

I really like this idea; the dependency on the HTTP framework is minimal, there's not much work in making it general. I suppose this general interface could be one call 'do POST request to URL with parameters X, get back status and raw data' which could be provided in (heck even curl works here as we use in the examples in bitcoind documentation).

[stevenroose]

I know you feel very strongly about hyper 0.10 vs hyper latest :) But yeah that sounds fine. I bet rust-jsonrpc-hyper (latest) and rust-jsonrpc-reqwest will be soon there though. Even though the idea of having a client implement the roundtripper interface should be quite trivial.

Another advantage of this is that the user can more easily tweak it's HTTP behavior like timeouts, proxy and could potentially add extra monitoring.

Lastly, I'd like to emphasize that there is no such thing as "bitcoin rpc" (cfr f.e. the Ethereum standardized RPC interface which is a huge mess). The RPC interface is Bitcoin Core-specific, so the name should reflect that.

I can take some time tomorrow morning to take a look at rust-jsonrpc. EDIT: Indeed a good opportunity to upgrade it to serde_json::Value. @apoelstra did you start some work on that already?

[apoelstra]

Good point. For sure, we should name our projects to reflect that we're using Bitcoin Core RPC and not some mythical "Bitcoin RPC".

[dpc]

Quick question. So should I keep adding to jeandudey/rust-bitcoin-rpc#11 until we have a crate etc.?

[apoelstra]

Yeah, I think that's the best strategy to keep moving forward quickly.

[dpc]

I've reached @jeandudey on twitter and he currently can't take care of his crates, so it's OK to move forward and work on them.

[jeandudey]

@dpc Is right I can't take care of my crates right now (I expect to be back in programming soon)

I think under those conditions we could reasonably pull both projects under rust-bitcoin, and also rust-bitcoin-rpc (assuming @jeandudey is ok with it). That crate is also probably where the rust-bitcoin Decimal and UDecimal types should live.

I'm ok with it, I think that these types should live there too.

[dpc]

@jeandudey Could you @apoelstra to crates.io owners of bitcoin-rpc? I think the crate will be renamed (bitcoincore-rpc?), but at least we could publish an update with README mentioning that the location changed.

[stevenroose]

@dpc did you take a look at https://github.com/stevenroose/rust-bitcoindrpc, btw? I don't know if you already did a lot of work on bitcoin-rpc, but it seems like I already implemented quite some things missing in bitcoin-rpc. like sane optional argument handling and using a single macro format for all calls. If having the json types in the same lib is a concern (which imo has not very much value), that's a minor change.

[dpc]

The whole list in jeandudey/rust-bitcoin-rpc#11. Major items below.

Changes

I've removed macros altogether. I feel very strongly that these macros are totally not worth it.

I've removed most of the allocations and proposed a do_rpc macro and some other improvements into the rust-jsonrpc itself: apoelstra/rust-jsonrpc#19, jeandudey/rust-bitcoin-rpc@8fcb2e8,
which makes these macros even less neccessary.

I've added a very simple example-test suite, that you can point at a node, and see if things still work.

I've changed error handling, so that error messages look better and contain method name: jeandudey/rust-bitcoin-rpc@c73b622

I've added a convenient get method that @apoelstra suggested: jeandudey/rust-bitcoin-rpc@637bdbb#diff-b4aea3e418ccdb71239b96952d9cddb6R150

I did bunch of maintaince (remove all unwrap, formating, deps. update, get rid of strason, move to serde_json).

Summing up

I'm fine with moving with your version @stevenroose, but let's have some second opinions about macros. For comparison https://github.com/dpc/rust-bitcoin-rpc/blob/dpc-dev/src/lib.rs#L334 doesn't look terrible, and (hopefully) doesn't scare people off.

Then we can discuss spliting types into separate crate (I think it's worth it, because it's very little work) and other changes that I've made one by one. :)

[stevenroose]

First and foremost, what is the issue with macros that makes you want to remove them?

So one of the things I focused on is optional argument handling. Arguments are put in order and if a call has 3 optional arguments, there is the scenario where the 2nd argument is given by the user so we should fill the default for the first, use the user's value for the second and skip the third.

One problem with trying to do this in a method, is that you can't create a vec![] with all your arguments, because the types don't match. I use a macro to all convert them with serde_json::to_value. Without the macro, that would mean passing every argument in a function first. It is possible of course.

[dpc]

First and foremost, what is the issue with macros that makes you want to remove them?

I'll quote:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

That's exactly what I think about DSLs and macros.

I get that a macro will help avoiding a little bit of repetition, but unless that repetition is something that the user has to do over and over, I don't see much value in it. Just copy, paste, modify and move on.

In https://github.com/stevenroose/rust-bitcoindrpc/blob/cdcae7148e27a446ab509d709b263a6dfa72e662/src/client.rs there's more lines implementing and commenting on the macro, than anything else, and as a newcomer, I have no idea what is going on there. I'll have to first learn the new DSL, just to add a small call that I might want to add. Also, probably some tools will fail to understand this macro and not generate autocompletion or eg. won't jump to definion of a function defined by this API, which is usability issue.

I haven't looked long, but I don't see anything in that macro that a function wouldn't be able to do. I'll try to convert listunspent into some non-macro version and see how it goes.

[stevenroose]

I pinged you on IRC. I did exactly that for two methods. One tiny thing I'm trying to get fixed and I'll share you.

[dpc]

I guess this can be closed?

[thomaseizinger]

I just wanted to mention that we did quite some work already on an RPC client as well: https://github.com/coblox/bitcoinrpc-rust-client

It currently depends on reqwest for sending the requests over the network.

Maybe we can make a dedicated crate for only the types + a trait that represents the rpc-interface of bitcoin-core? Since only the implementations need an actual HTTP client, we could probably get away with only serde as a dependency!

EDIT:
I just saw that this is exactly what's happening! 😅

[stevenroose]

@thomaseizinger Whoops, a wild third implementation appeared :)

Yeah so we consolidated the work of two separate clients together into one. It uses the jsonrpc crate, which currently uses hyper, but after apoelstra/rust-jsonrpc#20, it will be client-agnostic so you could write a reqwest-adaptor as well.

[thomaseizinger]

Yeah back when we started in April, there were 0 with the properties we were looking for (typesafe and without macros) :D

I'll take a look on what we can contribute to it! The most interesting things might be our testsuite (automatically spawns a new bitcoind instance via docker per test) and the auto-retry if bitcoind rpc interface is not yet ready (error status -32)!

[stevenroose]

@thomaseizinger That would be awesome! @dpc and I were talking about how we really need that! Let me know if I can help setting up Travis.

[dongcarl]

Closing, please discuss in https://github.com/rust-bitcoin/rust-bitcoincore-rpc