elementsproject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance

Core Lightning (CLN): A specification compliant Lightning Network implementation in C

Core Lightning (previously c-lightning) is a lightweight, highly customizable and standard compliant implementation of the Lightning Network protocol.

• Getting Started
  • Installation
  • Starting lightningd
  • Using the JSON-RPC Interface
  • Care And Feeding Of Your New Lightning Node
  • Opening A Channel
  • Sending and Receiving Payments
  • Configuration File
• Further Information
  • FAQ
  • Pruning
  • HD wallet encryption
  • Developers
• Documentation

Project Status

This implementation has been in production use on the Bitcoin mainnet since early 2018, with the launch of the Blockstream Store. We recommend getting started by experimenting on testnet (or regtest), but the implementation is considered stable and can be safely used on mainnet.

Any help testing the implementation, reporting bugs, or helping with outstanding issues is very welcome. Don't hesitate to reach out to us on IRC at #lightning-dev @ libera.chat, #c-lightning @ libera.chat, or on the implementation-specific mailing list [email protected], or on the Lightning Network-wide mailing list [email protected], or on Discord core-lightning, or on Telegram Core Lightning.

Getting Started

Core Lightning only works on Linux and macOS, and requires a locally (or remotely) running bitcoind (version 22.0 or above) that is fully caught up with the network you're running on, and relays transactions (ie with blocksonly=0). Pruning (prune=n option in bitcoin.conf) is partially supported, see here for more details.

Installation

There are 4 supported installation options:

• Installation of a pre-compiled binary from the release page on GitHub.
• Using one of the provided docker images on the Docker Hub.
• Compiling the source code yourself as described in the installation documentation.

Starting lightningd

Regtest (local, fast-start) Option

If you want to experiment with lightningd, there's a script to set up a bitcoind regtest test network of two local lightning nodes, which provides a convenient start_ln helper. See the notes at the top of the startup_regtest.sh file for details on how to use it.

. contrib/startup_regtest.sh

Mainnet Option

To test with real bitcoin, you will need to have a local bitcoind node running:

bitcoind -daemon

Wait until bitcoind has synchronized with the network.

Make sure that you do not have walletbroadcast=0 in your ~/.bitcoin/bitcoin.conf, or you may run into trouble. Notice that running lightningd against a pruned node may cause some issues if not managed carefully, see below for more information.

You can start lightningd with the following command:

lightningd --network=bitcoin --log-level=debug

This creates a .lightning/ subdirectory in your home directory: see man -l doc/lightningd.8 (or https://docs.corelightning.org/docs) for more runtime options.

Using The JSON-RPC Interface

Core Lightning exposes a JSON-RPC 2.0 interface over a Unix Domain socket; the lightning-cli tool can be used to access it, or there is a python client library.

You can use lightning-cli help to print a table of RPC methods; lightning-cli help <command> will offer specific information on that command.

Useful commands:

• newaddr: get a bitcoin address to deposit funds into your lightning node.
• listfunds: see where your funds are.
• connect: connect to another lightning node.
• fundchannel: create a channel to another connected node.
• invoice: create an invoice to get paid by another node.
• pay: pay someone else's invoice.
• plugin: commands to control extensions.

Care And Feeding Of Your New Lightning Node

Once you've started for the first time, there's a script called contrib/bootstrap-node.sh which will connect you to other nodes on the lightning network.

There are also numerous plugins available for Core Lightning which add capabilities: in particular there's a collection at: https://github.com/lightningd/plugins

Including helpme which guides you through setting up your first channels and customizing your node.

For a less reckless experience, you can encrypt the HD wallet seed: see HD wallet encryption.

You can also chat to other users at #c-lightning @ libera.chat; we are always happy to help you get started!

Opening A Channel

First you need to transfer some funds to lightningd so that it can open a channel:

# Returns an address <address>
lightning-cli newaddr

lightningd will register the funds once the transaction is confirmed.

Alternatively you can generate a taproot address should your source of funds support it:

# Return a taproot address
lightning-cli newaddr p2tr

Confirm lightningd got funds by:

# Returns an array of on-chain funds.
lightning-cli listfunds

Once lightningd has funds, we can connect to a node and open a channel. Let's assume the remote node is accepting connections at <ip> (and optional <port>, if not 9735) and has the node ID <node_id>:

lightning-cli connect <node_id> <ip> [<port>]
lightning-cli fundchannel <node_id> <amount_in_satoshis>

This opens a connection and, on top of that connection, then opens a channel. The funding transaction needs 3 confirmation in order for the channel to be usable, and 6 to be announced for others to use. You can check the status of the channel using lightning-cli listpeers, which after 3 confirmations (1 on testnet) should say that state is CHANNELD_NORMAL; after 6 confirmations you can use lightning-cli listchannels to verify that the public field is now true.

Sending and Receiving Payments

Payments in Lightning are invoice based. The recipient creates an invoice with the expected <amount> in millisatoshi (or "any" for a donation), a unique <label> and a <description> the payer will see:

lightning-cli invoice <amount> <label> <description>

This returns some internal details, and a standard invoice string called bolt11 (named after the BOLT #11 lightning spec).

The sender can feed this bolt11 string to the decodepay command to see what it is, and pay it simply using the pay command:

lightning-cli pay <bolt11>

Note that there are lower-level interfaces (and more options to these interfaces) for more sophisticated use.

Configuration File

lightningd can be configured either by passing options via the command line, or via a configuration file. Command line options will always override the values in the configuration file.

To use a configuration file, create a file named config within your top-level lightning directory or network subdirectory (eg. ~/.lightning/config or ~/.lightning/bitcoin/config). See man -l doc/lightningd-config.5.

A sample configuration file is available at contrib/config-example.

Further information

Pruning

Core Lightning requires JSON-RPC access to a fully synchronized bitcoind in order to synchronize with the Bitcoin network. Access to ZeroMQ is not required and bitcoind does not need to be run with txindex like other implementations. The lightning daemon will poll bitcoind for new blocks that it hasn't processed yet, thus synchronizing itself with bitcoind. If bitcoind prunes a block that Core Lightning has not processed yet, e.g., Core Lightning was not running for a prolonged period, then bitcoind will not be able to serve the missing blocks, hence Core Lightning will not be able to synchronize anymore and will be stuck. In order to avoid this situation you should be monitoring the gap between Core Lightning's blockheight using lightning-cli getinfo and bitcoind's blockheight using bitcoin-cli getblockchaininfo. If the two blockheights drift apart it might be necessary to intervene.

HD wallet encryption

You can encrypt the hsm_secret content (which is used to derive the HD wallet's master key) by passing the --encrypted-hsm startup argument, or by using the hsmtool (which you can find in the tool/ directory at the root of this repo) with the encrypt method. You can unencrypt an encrypted hsm_secret using the hsmtool with the decrypt method.

If you encrypt your hsm_secret, you will have to pass the --encrypted-hsm startup option to lightningd. Once your hsm_secret is encrypted, you will not be able to access your funds without your password, so please beware with your password management. Also, beware of not feeling too safe with an encrypted hsm_secret: unlike for bitcoind where the wallet encryption can restrict the usage of some RPC command, lightningd always needs to access keys from the wallet which is thus not locked (yet), even with an encrypted BIP32 master seed.

Developers

Developers wishing to contribute should start with the developer guide here.

lightning's Issues

Default config options not populated unless config file exists (debug-log branch)

I think the refactor (cd64932) that is currently the tip of the debug-log branch caused this. I'm not trying to fix it because it's 2AM here, and also I'm frankly a little scared of the config-parsing logic.

Use deterministic methods instead of randomness.

Eg. for onion pubkeys, we can use the hash of privkey etc. This would avoid entropy issues, but needs to be done carefully to avoid possible reuse.

Gossip Todo

This is a tracking issue for the gossip overhaul for the new architecture.

• Implement staggered broadcast for peers that are owned by gossipd
• Forward gossip messages to owner if gossipd is not the owner
• Create announcements for newly created channels
• Implement the getnodes JSON-RPC call
• Implement the getroute JSON-RPC call
• Implement the getchannels JSON-RPC call
• Add a new callback to wait for 6 confirmations before sending announcement_signatures
• Generate valid signatures for the announcements using the HSM #138

Depending on the decision on #126, we also need to

• Increase size of internal messages from u16 to u32 (might need some additional refactoring since currently we simply reuse the external wire format).

Minor issues in documentation

Hi Rusty,

I decided to finally lock myself away with your Reaching the Ground with Lightning paper and your earlier blog post series last night and spent some time digging into the details of Lightning. I love the work you're doing on this, and I'm happy that I'm finally beginning to get my head around lightning.

I have a few minor comments about the paper:

1. in Figure 1: Figure 1 from the Lightning Network Draft 0.5

   • Commitment Close Tx 1A (CC1b) -- should be CC1a

   • Commitment Close Tx 1b (CC1a) -- should be CC1b

     (both of these are carried over from the Lightning Paper 0.5 and fixed in 0.5.9)

2. in Figure 2: Figure 2 from the Lightning Network Draft 0.5

   • "Commitment Refund Tx1a (CR2a)" should be "Commitment Refund Tx2a (CR2a)"

   • The settlement/timeout transactions are named S1a/SD1a/T1a/TR1a. I believe these should be called S2a/SD2a/T2a/TR2a (since they're descendants of Commitment transaction 2a)

     (carried over from the Lightning Paper 0.5 and still in 0.5.9)

3. Section 3.2: "These can be later balanced by the lightning network itself, or an atomic-swap an on-chain bitcoin transaction" should be "These can be later balanced by the lightning network itself, or an on-chain atomic-swap transaction."
4. Section 3.2: "implementation splits fees where possible< and never allows either side" has an errant < character inserted (which doesn't appear in the .lyx file)
5. The references seem to have been broken between v0.1 and v0.2. References are mis-numbered and in the wrong order.
6. References to the Appendixes are broken ('The scripts for this can be found in 4' rather than 'The scripts for this can be found in appendix A' and 'as the commitment transaction in 4' instead of 'as the commitment transaction in appendix B')
7. In v0.2, the conclusion can be updated: 'By using a dual anchor and escape transactions, channel establishment can also avoid new CHECKSIG flags, though it loses the important ability to outsoure the enforcement of channel contract terms.' to 'By using a rebalanced single anchor, channel establishment can also avoid new CHECKSIG flags, while retaining the important ability to outsource the enforcement of channel contract terms.'
8. I'd recommend that the 'Escape transactions' and 'Fast escape transactions' in appendix A get moved to appendix B now that dual anchors are not part of the current design.

I'll open a pull request for 3 and 7. The others are either:

• changes to .eps files. I'm afraid I don't know how to modify those.
• errors in whatever tool you used to generate the .pdf from the source files.

I've also been following along on the Lightning-dev mailing list, and see that there's been a lot of movement on state machines and routing since v0.2 of the documentation. Are there plans to update the paper to include information on those aspects? I'm happy to lend a hand if you think that'd be useful.

Thanks,
John

lightningd creates 4000 child processes on startup

These seem to be created sequentially and so are harmless, other than slowing down startup and being annoying in gdb.

lightningd(27832): ID: xxx
Detaching after fork from child process 27836.
Detaching after fork from child process 27837.
Detaching after fork from child process 27838.
Detaching after fork from child process 27839.
...
Detaching after fork from child process 31926.
Detaching after fork from child process 31927.
Detaching after fork from child process 31928.
lightningd(27832): No fee estimate: using default fee rate
lightningd(27832): No fee estimate: using default fee rate
Detaching after fork from child process 31929.
lightningd(27832): Creating IPv6 listener on port 38377
lightningd(27832): Failed to bind on 2 socket: Address already in use
Detaching after fork from child process 31930.
lightningd(27832): Hello world!

Never produce dust outputs

http://lists.linuxfoundation.org/pipermail/lightning-dev/2015-November/000345.html

Missing OP_CSV in Figure 4 of the pdf

Hello,

I think there might be something missing in the htlc outputs of the Figure 4.

The payee should wait for a delay before being able to use the R, therefore the output would be:

R-VALUE & OP_CSV & SIG-PAYEE
OR
HTLC-TIMEOUT & SIG-PAYER
OR
REVOCATION-B & SIG A

Sorry I don't know how to make a PR for this, the file htlc.eps looks chinese to me !

Cheers,

Pierre

Cannot create a new channel after an existing channel closes

When trying to fund the new channel with a new tx, I get this error:

$ daemon/lightning-cli connect 159.203.218.14 6331 0100000001ae6aeaf3d1d5816434c2d68dc0fdd3f18abb8e59f4aa395e5129fc839704fceb0000000069463043021f3e7ef0460bea192e1364e65d40f4a94c807c8da1f5cc2335ec37bda953fdc10220678c5ca1e88e040bfe96c0b3b275d6b3cb55d6b4dd54bd67ed274f324a84fd32012103e4aea51a6b3bf1398135755e40c189016e10d7ae8d86093065fcb4376ae00374feffffff0240420f000000000017a914c195cfe511e78c115e5e8691b9f2d8520dedad2987dee23d05000000001976a9143f084e3f34e852b928abf7629ee9252b722e8d0888ac75740f00
"Already connected to peer 032282512a55842274728d58d371e1186a558071d6de509e32e5d6e765b0d3da7b"

This is the state of the peer:

        { "name" : "032282512a55842274728d58d371e1186a558071d6de509e32e5d6e765b0d3da7b:", "state" : "STATE_CLOSE_ONCHAIN_OUR_UNILATERAL", "peerid" : "032282512a55842274728d58d371e1186a558071d6de509e32e5d6e765b0d3da7b", "connected" : false, "our_amount" : 3924280000, "our_fee" : 67599999, "their_amount" : 0, "their_fee" : 1, "our_htlcs" : 
            [  ], "their_htlcs" : 
            [  ] }, 

lightningd can crash if time moves backwards

In the ccan timer module, there's an assertion that can crash if time moves backwards. I know there are different perspectives on this, but it seems like lightningd can probably recover from slight non-monotonicity of time, so it's worth thinking about whether crashing is the best outcome in this case.

lightningd: ccan/ccan/timer/timer.c:307: timers_expire: Assertion `now >= timers->base' failed.
lightningd(24325): FATAL SIGNAL 6 RECEIVED
Fatal signal 6. Log dumped in crash.log
Aborted

Crash when db has signature for state closing.

db:wrong bytes 0 not 64

Crash when trying to open a channel with amount below fee

crash.log

+514.907417361 lightningd(20657):jcon fd 7:DEBUG: Failing: Amount 1000 below fee 8972
+514.907478519 lightningd(20657):jcon fd 7:DEBUG: Failing: Amount 18446744073709543644 is too large
+514.907638050 lightningd(20657):BROKEN: FATAL SIGNAL 6 RECEIVED```

Silent failure to broadcast anchor tx, then crash

This time I tried 'connect'ing the cat picture server using an input tx that only had one confirm. I don't know whether that was the cause, but I ended up with a connection but no channel, and lightningd never said anything about broadcasting an anchor tx.

(First thing-that-seems-like-a-bug: If lightningd is not opening a channel because the tx I supplied isn't deep enough yet, it would be helpful if it would tell me -- this protocol is very fiddly so good error reporting is key. There wasn't anything in the log, either.)

At this point 'getpeers' reported:

{ "peers" : 
	[ 
		{ "name" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "state" : "STATE_OPEN_WAIT_FOR_OPEN_WITHANCHOR", "peerid" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706", "connected" : true } ] }

After a couple more confirms on the input tx, and nothing further from lightningd, I killed and restarted it. Still no anchor tx, but now 'getpeers' yielded:

{ "peers" : 
	[ 
		{ "name" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "state" : "STATE_INIT", "peerid" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706", "connected" : true } ] }

So I decided to run 'connect' again just to see what I would get, yielding finally:

lightningd(4399): Connected json input
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: Disconnected
lightningd: daemon/db.c:1686: db_forget_peer: Assertion `peer->state == STATE_CLOSED' failed.
lightningd(4399): FATAL SIGNAL 6 RECEIVED
Fatal signal 6. Log dumped in crash.log

So I guess I ended up in an inconsistent state and then went bang. That part definitely seems like a bug!

Build Error

daemon/json.c: In function ‘json_tok_u64’:
daemon/json.c:54:2: error: size of unnamed array is negative
BUILD_ASSERT(sizeof(l) >= sizeof(*num));
^
: recipe for target 'daemon/json.o' failed
make: *** [daemon/json.o] Error 1

listinvoice returns invalid JSON

$ daemon/lightning-cli listinvoice asdf
{ 
    [ 
        { "label" : "asdf", "rhash" : "4b9979d3836b9aa7e742f0d91de6648b6528502fad665a39505c7140d3a9605b", "msatoshi" : 1000, "complete" : false } ] }

The outer {} is both unnecessary and invalid, as JSON only allows key:value pairs inside {}.

process_estimatefee causes segfault

Seems bcli->args[3] should be bcli->args[2] on lines 194 and 206 of bitcoind.c.

My setup: 2 ubuntu VMs networked to eachother, one has bitcoind running, both have lightningd. lighning-cli connect [ip] [port] [tx] causes segfault on the machine that's connecting.

Can't reconnect to a peer after getting STATE_ERR_INFORMATION_LEAK

What kind of status is STATE_ERR_INFORMATION_LEAK?
I think this is occurred in case that closing a channel fails or peer breaks a channel forcedly according to https://github.com/ElementsProject/lightning/blob/e5b44ff23245267c0e54633acc9982ebe48c2a0c/daemon/state_types.h .

So I try to recreate a channel.
The connecting peer is booted successfully but connect command returns "peer breakdown".

How could I reconnect to the peer?

The peer side logs are like below.

IN-::ffff:1.55.46.187:21667: Connected in from ::ffff:1.55.46.187:21667 id 02d837e5fd98da358f8af239eedb7ea46d85400c088d10f018d3783c22800b1848, changing prefix
02c5bcc81e7953d90997be23f44f23f6b6086ddc239c26afc468ad2c1123085d07: Reconnected in from ::ffff:1.55.46.187:21579
02c5bcc81e7953d90997be23f44f23f6b6086ddc239c26afc468ad2c1123085d07: Sending PKT_ERROR: In error state STATE_ERR_INFORMATION_LEAK
02c5bcc81e7953d90997be23f44f23f6b6086ddc239c26afc468ad2c1123085d07: Disconnected
02c5bcc81e7953d90997be23f44f23f6b6086ddc239c26afc468ad2c1123085d07: Reconnected in from ::ffff:1.55.46.187:32081
02c5bcc81e7953d90997be23f44f23f6b6086ddc239c26afc468ad2c1123085d07: Sending PKT_ERROR: In error state STATE_ERR_INFORMATION_LEAK
02c5bcc81e7953d90997be23f44f23f6b6086ddc239c26afc468ad2c1123085d07: Disconnected

error in short channel id format ?

It seems that the 8-bytes short channel id format used in announcements is 4 bytes (block height) + 3 bytes (tx index) + 1 byte (tx output index), shouldn't it be 3+3+2 instead ?

Attempting to connect twice with same parameters crashes

When I try to connect a second time with the same parameters, I get this crash:

lightningd(22265): Connected json input
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: Disconnected
OUT-128.199.80.48:8334: Connected out to 128.199.80.48:8334 id 02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706, changing prefix
lightningd(22265): db_create_peer:constraint failed:INSERT INTO peers VALUES (x'02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706', 'STATE_INIT', 1, 221510);:UNIQUE constraint failed: peers.peer
Database error in peer_crypto_on
lightningd(22265): Database error in peer_crypto_on
Log dumped in crash.log

For my own reference if anybody wants to see it, this is crash.log.1474579521.

All attempts to open channels fail mysteriously

I'm not much help on the why of this one, but every attempt I make to open a channel is failing, here's an example.

The testnet bitcoind I'm talking to has been running for a long time, ignored by me, but appears to be up to date with the chain (it's on block 1012853, which matches some random explorer I looked at.)

(Note: This is not the first failed attempt I've made with this particular transaction, so it's possible that this is not the real failure, and this failure is actually "you already used this transaction" or "you already broke something with a previous failure", in which case the bug is that it doesn't say so.)

./daemon/lightning-cli connect 128.199.80.48 8334 010000000108de68e965b8b89d3faacbb8be39f98bbe8beec84645ee834fbd5b85df79005a010000006b48304502210090db29ed1b2b7555b94bf863ece893fc588fe6632ed789c2042a42f68071bcfd022069cc5884d20409d8abf339fb1438a423c483ea059a5aa5b5f67f1789200bcb72012103417a853d089af430f8ec6593d94daa7d98842f467ff1439bf8ef6ecae59f1aebfeffffff0240420f000000000017a914e13ebe92e2e2290c88497a7d8e635485217733a68738700105000000001976a9147ad3816abe25f6932738a9a7d65d681c10db3c9b88ac0d740f00
[hangs]

lightningd says:

OUT-128.199.80.48:8334: Connected out to 128.199.80.48:8334 id 02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706, changing prefix
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: Entered error state STATE_ERR_INTERNAL
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: Peer breakdown: nothing to do
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: Disconnected

getlog debug says:

{ "type" : "INFO", "time" : "99.399146956", "source" : "OUT-128.199.80.48:8334:", "log" : "Connected out to 128.199.80.48:8334 id 02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706, changing prefix" }, 
    { "type" : "DEBUG", "time" : "99.399173959", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "Using fee rate 252000" }, 
    { "type" : "DEBUG", "time" : "99.399516067", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "db_create_peer(02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706)" }, 
    { "type" : "DEBUG", "time" : "99.399524245", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "db_start_transaction(02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706)" }, 
    { "type" : "DEBUG", "time" : "99.399757572", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "db_commit_transaction(02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706)" }, 
    { "type" : "DEBUG", "time" : "99.408130983", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "Queued pkt PKT__PKT_OPEN (order=0)" }, 
    { "type" : "DEBUG", "time" : "99.408137194", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "CMD_OPEN_WITH_ANCHOR: STATE_INIT => STATE_OPEN_WAIT_FOR_OPEN_WITHANCHOR (out PKT__PKT_OPEN)" }, 
    { "type" : "DEBUG", "time" : "99.408148105", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "pkt_out: writing PKT__PKT_OPEN" }, 
    { "type" : "DEBUG", "time" : "99.615157245", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "Decrypted header len 5" }, 
    { "type" : "DEBUG", "time" : "99.615197380", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "Received packet LEN=5, type=PKT__PKT_RECONNECT" }, 
    { "type" : "DEBUG", "time" : "99.615203960", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "CMD_SEND_COMMIT: STATE_OPEN_WAIT_FOR_OPEN_WITHANCHOR => STATE_ERR_INTERNAL" }, 
    { "type" : "BROKEN", "time" : "99.615214327", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "Entered error state STATE_ERR_INTERNAL" }, 
    { "type" : "INFO", "time" : "99.615229732", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "Peer breakdown: nothing to do" }, 
    { "type" : "DEBUG", "time" : "99.615235671", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "peer_breakdown: STATE_ERR_INTERNAL => STATE_CLOSED" }, 
    { "type" : "DEBUG", "time" : "99.615247719", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "db_forget_peer(02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706)" }, 
    { "type" : "DEBUG", "time" : "99.615261653", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "db_start_transaction(02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706)" }, 
    { "type" : "DEBUG", "time" : "99.615714001", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "db_commit_transaction(02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706)" }, 
    { "type" : "DEBUG", "time" : "99.620063221", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "pkt_out: no IO possible, closing" }, 
    { "type" : "INFO", "time" : "99.620095618", "source" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "log" : "Disconnected" }, 

Assertion when automatically closing channel

On both sides:

lightningd: daemon/peer.c:165: bitcoin_commit: Assertion `!peer->local.commit->tx->input[0].witness' failed.
lightningd(29329): FATAL SIGNAL 6 RECEIVED
Fatal signal 6. Log dumped in crash.log
Aborted (core dumped)

035f opened the channel.
crash_segwit_035f.txt
crash_segwit_0218.txt

Just want to say 'Thank You'.

Thanks for the work you're putting into this, it's very much needed. And although you're using C, you seem to have managed to avoid C++. I guess thank you as well for that. :)

Another reason for declining an htlc

Hello,

Currently the only available ones are insufficient_funds and cannot_route.

But since the commitment tx must be publishable at any time, we might need to decline a new htlc because that would make the tx too big.

Should we introduce a new reason too_many_pending_htlc?

Thanks

Pierre

We are getting kicked off Freenode

I was contacted by a Freenode staff member, informing me that the channel will get kicked off the network at the end of the week.

Build Error

cc -Werror -Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition -g -fstack-protector -I ccan -I secp256k1/include/ -I . -c -o daemon/peer.o daemon/peer.c
daemon/peer.c: In function ‘their_htlc_added’:
daemon/peer.c:943:10: error: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 2 has type ‘u64 {aka long long unsigned int}’ [-Werror=format=]
printf("FORWARDING %lu\n", step->hoppayload->amount);
^
cc1: all warnings being treated as errors
: recipe for target 'daemon/peer.o' failed
make: *** [daemon/peer.o] Error 1

Can't close channels in state STATE_ERR_INFORMATION_LEAK

lightning-cli close 0297377f0d4c18c61a451b8d57525a305f92d3d74638891e6eadf4f3c82c66dc8e
"Peer is already closing: state STATE_ERR_INFORMATION_LEAK"

This causes that peer to never be able to connect to me again:

OUT-159.203.218.14:6331: Connected out to 159.203.218.14:6331 id 032282512a55842274728d58d371e1186a558071d6de509e32e5d6e765b0d3da7b, changing prefix
032282512a55842274728d58d371e1186a558071d6de509e32e5d6e765b0d3da7b: reconnect ack 4 > 0
032282512a55842274728d58d371e1186a558071d6de509e32e5d6e765b0d3da7b: Sending PKT_ERROR: invalid ack
032282512a55842274728d58d371e1186a558071d6de509e32e5d6e765b0d3da7b: Peer breakdown: nothing to do
032282512a55842274728d58d371e1186a558071d6de509e32e5d6e765b0d3da7b: Disconnected

Build fails with: lightningd/pay.c:1:17: fatal error: pay.h: No such file or directory

$ cd lightning
$ make
...
fstack-protector -I ccan -I libwally-core/src/secp256k1/include/ -I libwally-core/include/ -I libsodium/src/libsodium/include/ -I . -DSHACHAIN_BITS=48 -c -o lightningd/pay.o lightningd/pay.c
lightningd/pay.c:1:17: fatal error: pay.h: No such file or directory
compilation terminated.

lightningd/pay.c:1: #include "pay.h"

$ git ls-files | grep pay.h
daemon/pay.h

Can't do multi-hop micropayment

I try to do multi-hop micropayment.
Lightningd already implemented multi-hop?

$ ./lightning/daemon/lightningd --version
v0.5.2-2016-11-21-146-g1c49af7
aka. Bitcoin Savings & Trust Daily Interest II
Built with:

crash due to memory corruption when opening channel

I have two nodes, with a channel opened from node A to node B.
When I try to create another channel from node B to node A, the node B crashes on memory corruption.
Version is current head, v0.5.2-2016-11-21-412-g018c1d9

*** Error in `daemon/lightningd': malloc(): memory corruption: 0x00000000038c4940 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f2d2ee6a7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8181e)[0x7f2d2ee7481e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f2d2ee765d4]
daemon/lightningd[0x4599f1]
daemon/lightningd[0x459fa7]
daemon/lightningd[0x435b75]
daemon/lightningd[0x435c95]
daemon/lightningd[0x415e52]
daemon/lightningd[0x44e1ab]
daemon/lightningd[0x44ece6]
daemon/lightningd[0x4501b7]
daemon/lightningd[0x450242]
daemon/lightningd[0x4034ec]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f2d2ee13830]
daemon/lightningd[0x403039]

Stacktrace is not particularly helpful, so I have attached a valgrind log that shows read and write to already free'd memory.

valgrind.txt

Build fails

lightningd/pay.c:1:17: fatal error: pay.h: No such file or directory
compilation terminated.

Small typo in Makefile

There's a small typo in the main Makefile: -DHAS_CLTV=1 is now mispelled -DHAS_CTLV=1

close on non-NORMAL channel causes assert

From @cedcker comment in #67

https://gist.github.com/anonymous/251ce28774d9f0c094a5dbe06388b6bb

Turns out it was a bit different. calling close on a channel that is neither in STATE_NORMAL nor STATE_NORMAL_COMMITTING will trigger an assertion and fault our client. But it seems that we are conflating connection state and channel state in a few places.

Don't use u32 locktime internally.

We use it for both relative timeouts, where it's actually used in nSequence and OP_CSV, and absolute timeouts in locktime. Over the wire it's represented explicitly, but converting to locktime then BIP68 nSequence is just weird.

32-bit machines make check fails

$ daemon/test/test-routing --verbose
...
lightningd: Computed MAC does not match expected MAC, the message was modified.
lightningd: Computed MAC does not match expected MAC, the message was modified.
lightningd: Computed MAC does not match expected MAC, the message was modified.

Error starting lightningd

Not sure what the error is but whenever I try to start lightnind the following error is thrown:

DATABASE NEEDS UPDATE.  Can't access VERSION: SQL logic error or missing database:no such table: version
lightningd(18306): DATABASE NEEDS UPDATE.  Can't access VERSION: SQL logic error or missing database:no such table: version
Log dumped in crash.log

Tried troubleshooting by reinstalling the libsqlite library but didnt worked.
bitcoind is configured for testnet and works normally.
OS: Ubuntu 16.04 LTS

For reference tried looking up for crash.log but unable to find its location.

Can't lookup invoices with numeric labels with lightning-cli

This also reproduces over other json-rpc clients if you fail to cast label to a string.

$ daemon/lightning-cli listinvoice
{ 
    [ 
        { "label" : "1", "rhash" : "91c45480a2047159de6bc01c1cf8dfd4d4a54f2cdd41cf9528baa229b940cc10", "msatoshi" : 1, "complete" : false }, 
        { "label" : "asdf", "rhash" : "4b9979d3836b9aa7e742f0d91de6648b6528502fad665a39505c7140d3a9605b", "msatoshi" : 1000, "complete" : false } ] }

$ daemon/lightning-cli listinvoice asdf
{ 
    [ 
        { "label" : "asdf", "rhash" : "4b9979d3836b9aa7e742f0d91de6648b6528502fad665a39505c7140d3a9605b", "msatoshi" : 1000, "complete" : false } ] }

$ daemon/lightning-cli listinvoice 1
{ 
    [  ] }

Install instructions for debian?

Should I be able to use install on debian similar to ubuntu? I'm having issues getting the libbase58-dev package, apparently it isn't for debian wheezy (latest version). Do I need to re-configure sources or something to get that package? thanks

'getroute' documentation and error message need updating

The 'getroute' documentation and error message only mention two parameters, but it actually takes three (they were not updated when the third one was added, or else the third one is supposed to be optional and is not.)

Also the cat picture instructions need to be updated.

connect hangs if given an already-spent raw tx

lightningd gives the following error:

lightningd(12839): Broadcasting tx 020000000001015cdd29ac75ead0492ad22a4820f62abeef184f5fcefd50dff5aee9921ea6897d0000000017160014730569fc932bc0c9bd44e7a834d46bc938df4e61ffffffff0148e93c000000000022002009d9aa2eb572be5e33eea84f1e2241c88d9d9a8fb0665f79e78290eb880d271b0247304402205f7cc0f82bc6195ffefce0cfbd4daab62532074b6a09ca4513d91ba9603c3ffb02202afba691351b368f16b1a8e1d7f9597af33f9ab4f4768fb36cf2ef8988a12d3f01210318f97a9bce8e4a2c4cca0508219bf5e8839db02448572c729a22bb7c9198497000000000: error code: -26
error message:
18: bad-txns-inputs-spent

The connect json-rpc call should probably return right away and report this error to the user.

open_anchor supplies txhash instead of txid

Hello,

This is just a matter of reversing the array, but the comment is a bit misleading

message open_anchor {
  // Transaction ID of anchor.
  required sha256_hash txid = 1;

See https://github.com/ElementsProject/lightning/blob/master/bitcoin/tx.c#L232

Cheers,

Pierre

'connect' needs better error reporting

When I run this:

gwillen@lightning-test:~/src/lightning$ ./daemon/lightning-cli connect 128.199.80.48 8334 01000000015bd0cf9bd023ad00f682ce8999c6f0dc750b1f3655824889aad1f235d8787570010000006b48304502210095448f70e829fd2d0268d1230eb44edfab8f657cfb2a9a4efd21cfb4313a3faa022021f099a54676482e7bdcbb6a132f331a74302289c47d0b014b96f565317d2ded012102116f21a0310ea54d54a7960985dfccc09f069caa273f4d55beb3459366092865feffffff02409ef629010000001976a9147d48e46fcc3e186118ee57f28ea3edc263e503ce88ac40420f000000000017a91470ecbf2dca0c46424fa8f613d435e54ed5688e8287ea2c0e00

I get only this output:

{ }

But in the log I get this:

OUT-128.199.80.48:8334: Connected out to 128.199.80.48:8334 id 02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706, changing prefix
lightningd(9867): No fee estimate: using default fee rate
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: Sending PKT_ERROR: Commitment fee rate too low
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: peer open failed: Commitment fee rate too low
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: Peer breakdown: nothing to do
02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706: Disconnected
lightningd(9867): Unable to estimate fee

Per @rustyrussell, the output should contain a more useful error message.

Stuck transactions after initial send

Payments to the 0218 peer appear to be stuck.

12MB logs from dev-crash:
https://people.xiph.org/~tdaede/crash_035f.log
https://people.xiph.org/~tdaede/crash_0218.log

View from 035f:

$ daemon/lightning-cli getpeers
{ "peers" : 
    [ 
        { "name" : "0218e799f5085d90716493c0f9b5b7373ae47a0fb0c8cfba7411786f216e0ed15a:", "state" : "STATE_NORMAL", "peerid" : "0218e799f5085d90716493c0f9b5b7373ae47a0fb0c8cfba7411786f216e0ed15a", "connected" : false, "our_amount" : 1924280000, "our_fee" : 67599999, "their_amount" : 0, "their_fee" : 1, "our_htlcs" : 
            [ 
                { "msatoshi" : 1, "expiry" : 
                    { "block" : 1011980 }, "rhash" : "a957ac4630cf76cdc03957789b36a145665e64ef1a18d2e6a5a23f56e899e17d", "state" : "SENT_ADD_HTLC" }, 
                { "msatoshi" : 1, "expiry" : 
                    { "block" : 1011980 }, "rhash" : "9e8a2bcb36e754eb287db677655fe783c914ac90eff276e3ba9fb8a56f0b2b4b", "state" : "SENT_ADD_HTLC" } ], "their_htlcs" : 
            [  ] }, 
        { "name" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706:", "state" : "STATE_NORMAL", "peerid" : "02915506c736ffec49ad58fc021779600dcd2b7a52ac97690571aea5b4d9be2706", "connected" : true, "our_amount" : 3954660000, "our_fee" : 40378000, "their_amount" : 0, "their_fee" : 100000, "our_htlcs" : 
            [  ], "their_htlcs" : 
            [  ] } ] }

View from 0218:

{ "peers" : 
    [ 
        { "name" : "035f4dafaf485d5b3245a784cc02587e745d450d15eaaf187cf68a4f149569c522:", "state" : "STATE_NORMAL", "peerid" : "035f4dafaf485d5b3245a784cc02587e745d450d15eaaf187cf68a4f149569c522", "connected" : false, "our_amount" : 0, "our_fee" : 1, "their_amount" : 1924280000, "their_fee" : 67599999, "our_htlcs" : 
            [  ], "their_htlcs" : 
            [  ] } ] }

Removing the legacy daemon

Since we have almost reached feature parity it's time to remove the legacy daemon from the source tree. I think the following features are missing before we can do that:

• Support re-establishing connections on disconnect
• Support re-establishing connections after restarts
• Track funds across restarts
• Closing negotiation
• On-chain unilateral close
• On-chain penalty transaction

This being a rather large change, I would also propose a reorganization of the source tree:

• Remove source files that are not in use by the new daemon
• Remove protobuf-c as a dependency
• Rename the lightningd directory to src, and move all other source directories into that directory. This results in a clean top level directory with README.md with associated documentation, and the contrib directory.

Did I miss something @rustyrussell?

asciidoc required to build

On my freshly-installed Debian unstable system, I have to apt-get install asciidoc (not listed in the instructions) in order to get the 'a2x' binary which is required by some part of the lightning build.

Bad signature on sendpay

There seems to be a regression on the debug-log branch that causes sendpay to fail with Bad Signature. This is a 1-hop test payment on testnet. The receiving node apparently provides an invalid signature when returning the commit transaction, or the sender validates the signature incorrectly.

The problem was not there in the master branch. Currently bisecting to find where the bug was introduced.

Dockerfile and other updates

Issue

• The Dockerfile is based on Ubuntu which tends to be a bit heavy on the storage.
• Multiple installation over different instances of Lighting can be time-consuming processes.

Solution

Hence as a next step forward from the issue #167 maybe the following should be addressed:

• Update the Dockerfile to Alpine or similar distribution
• Automated script for 'one-click' installation of LN and its dependencies

Not exactly issues but feel it needs to be addressed for make it easier for people to play with the implementation.

P.S. Was unable of find the correct title :P

getroute to myself crashes lightningd

$ daemon/lightning-cli getroute 035f4dafaf485d5b3245a784cc02587e745d450d15eaaf187cf68a4f149569c522 1000 1
lightning-cli: Non-object response ''

Backtrace:

#0  0x0000000000429c51 in find_route (dstate=0x697030, to=0x7fffffffdae0, msatoshi=1000, 
    riskfactor=1, fee=0x7fffffffdaa0, route=0x7fffffffdab0) at daemon/routing.c:345
        n = 0x0
        src = 0x698510
        dst = 0x698510
        it = {i = {off = 4}}
        first = 0xa321d10
        runs = 20
        i = 21
        best = 0
        __PRETTY_FUNCTION__ = "find_route"
#1  0x0000000000419eda in json_getroute (cmd=0x722120, 
    buffer=0xa321c80 "{ \"method\" : \"getroute\", \"id\" : \"lightning-cli-4156\", \"params\" : [ \"035f4dafaf485d5b3245a784cc02587e745d450d15eaaf187cf68a4f149569c522\", 1000, 1] }", 
    params=0x9deebb0) at daemon/pay.c:244
        id = {pubkey = {
            data = "\"\305i\225\024O\212\366|\030\257\352\025\rE]t~X\002̄\247E2[]H\257\257M_C\025w\245\071\025\327R\231eyӇ3\234KX\276\214\223\316R\324Q\314\344lD\233\337\017\245"}}
        idtok = 0x9deebc0
        msatoshitok = 0x9deebd0
        riskfactortok = 0x9deebe0
        response = 0x42fe7a <json_tok_streq+137>
        i = 21
        msatoshi = 1000
        fee = 140737488345840
        riskfactor = 1
        route = 0x44daa5 <list_add_+44>
        peer = 0x14
        amounts = 0x1
        total_amount = 1
        total_delay = 0
        delays = 0xa321ca0
#2  0x0000000000415c3c in parse_request (jcon=0xa769710, tok=0x9deeb50) at daemon/jsonrpc.c:492
        method = 0x9deeb70
        id = 0x9deeb90
        params = 0x9deebb0
        cmd = 0x466680 <getroute_command>
        __PRETTY_FUNCTION__ = "parse_request"
#3  0x0000000000415ee1 in read_json (conn=0x5513fd0, jcon=0xa769710) at daemon/jsonrpc.c:554
        toks = 0x9deeb50
        valid = true
#4  0x0000000000443255 in next_plan (conn=0x5513fd0, plan=0x5514018) at ccan/ccan/io/io.c:58
        next = 0x415d83 <read_json>
        __PRETTY_FUNCTION__ = "next_plan"
#5  0x0000000000443cda in do_plan (conn=0x5513fd0, plan=0x5514018) at ccan/ccan/io/io.c:375
        __PRETTY_FUNCTION__ = "do_plan"
#6  0x0000000000443d18 in io_ready (conn=0x5513fd0, pollflags=1) at ccan/ccan/io/io.c:386
No locals.
#7  0x0000000000444fd4 in io_loop (timers=0x6970d8, expired=0x7fffffffdcf0)
---Type <return> to continue, or q <return> to quit---
    at ccan/ccan/io/poll.c:303
        c = 0x5513fd0
        events = 1
        i = 4
        r = 0
        ms_timeout = 6561
        ret = 0x697030
        __PRETTY_FUNCTION__ = "io_loop"
#8  0x000000000040477c in main (argc=1, argv=0x7fffffffde08) at daemon/lightningd.c:529
        expired = 0x0
        v = 0x0
        dstate = 0x697030
        portnum = 0

Nodes do not announce newly established channels in IRC

Currently nodes only announce on a very long timer or when node joins the channel. If a new channel is established the endpoints will not announce it right away, instead they'll wait for one of the above triggers.

Signature checking is too weak

@sipa points out:

sipa: you need a slightly stronger check
sipa: the low-s requirement is that s is below n/2
sipa: and n is around 2^256 - 2^128

You could screw the current code up by making me think you had given me a valid sig, but actually hadn't, and I'd lose.

Add routing information gossip

This is a tracking issue for the replacement of the IRC based node and channel announcement with a peer-to-peer based gossiping protocol.

• Add nested transport in protobuf-based protocol
• Add message for node and channel announcement
• Trigger timeout based announcements
• Reflect incoming announcements in routing table
• Staggered forward of incoming announcements
• Trigger channel event based announcements
• Dump channel_announcements and channel_updates to newly connected peers

I'm currently developing this locally, and will file a PR as soon as it is ready to review.

Closing w/ unacked htlcs

Hi,

I think there is something wrong with how unacked htlcs are handled by lightning-c during mutual closing. The spec is a little vague on this so I can't tell if lightning-c implementation does or doesn't comply. I guess it depends on how one side indicating that it wants to clear the channel (and thus will accept no new HTLCs) (bolt02, §4) is interpreted.

In file peer.c (lines 402 and 434), we go to STATE_MUTUAL_CLOSING when there are no htlcs in signed tx (actual check is done in function committed_to_htlcs).

But I don't see why we shouldn't keep the unacked htlcs. As soon as both parties have sent and received a close_clearing packet, and because they agree not to send any new htlcs after having sent their own close_clearing packet (bolt02, §4.1), the state is perfectly well-defined and all pending htlcs (sent & received) could be included.

What bothers me is that in the rest of the protocol we assume that a valid htlc will always be accepted (there is no way for a peer to say 'no'), so the fact that in this specific case some htlcs could be discarded doesn't seem very consistent.

What do you think?

Cheers,

Pierre