Make `cln-grpc` a first-class citizen about lightning

[rustyrussell]

My preference is to simplify APIs where necessary: the GRPC can simply use the new APIs so the only deprecation cycle is on the JSON-RPC side, and I can handle those.

Can we make a checklist of what APIs are missing? GH supports those natively, so we can use this issue to coordinate and aim to complete that by next release? (Sure, that means we need it done in three weeks, but pressure makes diamonds, right?)

[ohenrik]

I seconds the importance of this. While Building Torq and using CLN for Eden we notice that the biggest weakness that CLN has is the APIs and the gRPC. I think that improving this will have a massive impact on CLN adoption especially for enterprise.

Another thing. Not having subscription for certain data forces the need to build plugins. For example for payment status and channel interception.

[daywalker90]

I think these belong here aswell:

#7185
#7186

[daywalker90]

Can we make a checklist of what APIs are missing?

Here you go #7209 @rustyrussell - now even tests can catch missing calls

I have commits ready for all of them except sql/sql-template

[cdecker]

Thanks @kilrau for taking the lead here. cln-grpc is a core component of CLN and Greenlight alike, and so I am interested in getting coverage and stability to the same level as the other client bindings too.

The JSON-RPC schema, from which the GRPC bindings are generated via msggen, are sometimes rather complex, meaning that the msggen converter can't handle them all as of now. We can either simplify the schema (my preference, but also slow due to deprecation cycles) or we can teach msggen how to handle these more complex structures.

So we'd be attacking this on 3 fronts:

• JSON-RPC Schema must be simplified to match the capabilities of other RPC systems we intend to use. GRPC for now.
• msggen needs to learn how to deal with the structures we have already
• Testing needs to be expanded to allow testing through the grpc interface too.

The last point already partially exists, and is what I used when building msggen for my own needs. If you set the envvar CLN_TEST_GRPC=1 you'll switch to using the cln-grpc interface instead of the direct JSON-RPC:

lightning/contrib/pyln-testing/pyln/testing/utils.py

Lines 831 to 835 in e331532

	if os.environ.get('CLN_TEST_GRPC') == '1':
	logging.info("Switching to GRPC based RPC for tests")
	self._create_grpc_rpc()
	else:
	self._create_jsonrpc_rpc(jsonschemas)

Technically, we have grpc2py.py which translates requests on the python side into the corresponding grpc messages, then sends them to the cln-grpc interface, and then it translates back into the JSON-RPC format so the tests don't have to change. This roundtripping could be a good way to ensure both interfaces remain stable.

What do you think?

[kilrau]

So first step is to fix msggen. @michael1011 gave it a look - it honestly looks like someone from your team who's already familiar with it will need to do that, too steep of a learning curve for an external contributor. I can't judge if simplifying the schema is currently possible or teaching msgen to handle the existing structures, but who from your team could work on it?

[kilrau]

New issues since I opened this one:
#7168
#7169
#7176

[NicolasDorier]

Just noting that in my case, the fact it is a plugin let me keep supporting arm32v7 due to this annoying issue. #6596

The fact I can remove it easily from the build in our fork fixed my issue. (I spent 1 day trying to fix it)

[Impa10r]

Don't want to start a new discussion, but hope for experts here to answer my question. Can gRPC methods be added dynamically at runtime for a CLN plugin? Looking for ways to make my PeerSwap Web compatible with Core Lightning.

[kilrau]

Can we make a checklist of what APIs are missing?

Here you go #7209 @rustyrussell - now even tests can catch missing calls