keep-network / keep-common

Common libraries and tools used across Keep repositories

keep-common

Common libraries and tools used across Keep repositories.

Directory structure

The directory structure used in the keep-common repository is the same as used on other Keep repositories, and very similar to that used in other Go projects:

keep-common/
  tools/ (1)
    generators/ (2)
  pkg/ (3)
    chain/
      chain.go, *.go (4)
      ethereum/
        gen/
          gen.go (5)
    relay/
      relay.go, *.go

1. Keep tools have categorized directories here, which are typically Go commands that can be used from other repositories.

2. Code generators in particular live in this subtree.

3. All additional Go packages live in pkg/.

4. The high-level interfaces for a package mypackage live in mypackage.go. Packages like chain are interface packages that expose a common interface to network and blockchain layers, for example. Their subpackages provide particular implementations of these common interfaces, if these are considered shared implementations..

5. When a package requires generated code, it should have a subpackage named gen/. This subpackage should contain a single file, gen.go, with a // go:generate annotation to trigger appropriate code generation. All code generation is done with a single invocation of go generate at build time.

Installation

• Clone this repo

• Install go v1.18: $ brew install [email protected]

• Generate go files: $ go generate ./…​/gen

• Build the project: $ go build ./…​

• Run tests: $ go test ./…​

keep-common's Issues

Possibly incorrect gas estimate for payable functions

In #24 we introduced a function allowing to estimate gas manually before submitting transaction. We are not passing Value to CallMsg in EstimateGas function of ethutil.go so the estimation may not be correct for payable functions.

Code generation does not work with abigen 1.9.18-stable

After upgrading abigen from 1.9.10-stable to 1.9.18-stable I can no longer generate contracts code:

/Users/piotr/go/pkg/mod/github.com/keep-network/[email protected]/tools/generators/ethereum/contract_parsing.go:154:12: method.Const undefined (type abi.Method has no field or method Const)
/Users/piotr/go/pkg/mod/github.com/keep-network/[email protected]/tools/generators/ethereum/contract_parsing.go:167:24: param.Type.Type undefined (type abi.Type has no field or method Type)
/Users/piotr/go/pkg/mod/github.com/keep-network/[email protected]/tools/generators/ethereum/contract_parsing.go:202:26: output.Type.Type undefined (type abi.Type has no field or method Type)
/Users/piotr/go/pkg/mod/github.com/keep-network/[email protected]/tools/generators/ethereum/contract_parsing.go:215:42: method.Outputs[0].Type.Type undefined (type abi.Type has no field or method Type)
/Users/piotr/go/pkg/mod/github.com/keep-network/[email protected]/tools/generators/ethereum/contract_parsing.go:232:12: method.Const undefined (type abi.Method has no field or method Const)
/Users/piotr/go/pkg/mod/github.com/keep-network/[email protected]/tools/generators/ethereum/contract_parsing.go:251:24: param.Type.Type undefined (type abi.Type has no field or method Type)
make: *** [contract/TokenStaking.go] Error 2
pkg/chain/gen/contract.go:3: running "sh": exit status 2

Consider a transaction resubmitter for gas price adjustment

Consider providing a subcommand that, when invoked with a transaction hash originally signed by the current key, would resubmit the same transaction with a higher gas price.

Adding package level doc for `encryption` and `persistance`

Contract binding generator: Wrong handling of multiple return values

During the work on keep-network/keep-core#3427, we noticed a problem with a generated contract binding that maps the WalletRegistry contract. That contract defines the following function:

function isDkgResultValid(DKG.Result calldata result) external view returns (bool, string memory)

which is covered by the WalletRegistry.go binding as:

func (wr *WalletRegistry) IsDkgResultValid(
    arg_result abi.EcdsaDkgResult,
) (isDkgResultValid, error) {
    ...
}

type isDkgResultValid struct {
	bool
	string
}

The isDkgResultValid is package-private and contains anonymous fields that can't be accessed outside. The only possible workaround is by using Go's reflection. This should be improved by making the returned values accessible.

Intermittent Metrics Output

I've observed intermittent errors when collecting metrics. Metrics will disappear from the output, I seems to have remedied the issue by dropping the tick timer for Ethereum down to 60 from the default 600 but I couldn't see from the code why this was happening

Steps

• Run a node (either core or ecdsa)
• Enable metrics with default settings
• Scrape the /metrics endpoint

Expected

Consistent output for all metrics

Observed

Metrics periodically drop from output, specifically observed have been eth_connectivity and libp2p_info

Enable continuous integration on keep-common

For now, just running Go generation and tests should be enough.

Contract binding generator: handle conflicting struct declarations

May depend on #45

See keep-network/keep-core#3630

after_abi_hook in keep-core renames structures that were incorrectly re-declared in the same Go package given this bug: ethereum/go-ethereum#24627.

The generated tbtc/gen/abi/Bridge.go, tbtc/gen/abi/WalletCoordinator.go, and tbtc/gen/abi/MaintainerProxy.go declare BitcoinTxInfo struct. To make the Go compiler happy, we rename BitcoinTxInfo to BitcoinTxInfo2 and BitcoinTxInfo3.

Those Go files are generated with @go run github.com/ethereum/go-ethereum/cmd/abigen.

But this is not enough! If we try to build the project with just this hook, we will encounter errors like:

pkg/chain/ethereum/tbtc/gen/contract/MaintainerProxy.go:730:3: cannot use arg_mainUtxo (variable of type "[github.com/keep-network/keep-core/pkg/chain/ethereum/tbtc/gen/abi](http://github.com/keep-network/keep-core/pkg/chain/ethereum/tbtc/gen/abi)".BitcoinTxUTXO) as type "[github.com/keep-network/keep-core/pkg/chain/ethereum/tbtc/gen/abi](http://github.com/keep-network/keep-core/pkg/chain/ethereum/tbtc/gen/abi)".BitcoinTxUTXO2 in argument to mp.contract.NotifyMovingFundsBelowDust

Looking at the generated code at the problematic line:

transaction, err := mp.contract.NotifyMovingFundsBelowDust(
  transactorOptions,
  arg_walletPubKeyHash,
  arg_mainUtxo,
)

mp.contract.NotifyFundsBelowDust is declared in the tbtc/gen/abi/MaintainerProxy.go and expects BitcoinTxUTXO2.

arg_mainUtxo from tbtc/gen/contract/MaintainerProxy.go has an incorrect type though: arg_mainUtxo abi.BitcoinTxUTXO,

Just like tbtc/gen/abi/MaintainerProxy.go this file is generated based on tbtc/gen/abi/MaintainerProxy.abi and in the *.abi file we don't have the rename performed by after_abi_hook.

The compilation error was solved with the after_contract_hook which is partially a result of ethereum/go-ethereum#24627 and partially how our Go contract bindings work. The fix from ethereum/go-ethereum#24924 will change the generated tbtc/gen/abi/*.go files but we'll have to adjust the @go run github.com/keep-network/keep-common/tools/generators/ethereum logic to follow the same naming rules as the *.abi file will remain unchanged.

Adjust generators to be externally usable

Two target generators here:

• pkg/gen contains the generator for promises. Move to tools/generators/async and make it so we can reference this from other packages.
• pkg/chain/gen contains the generator for contract glue as well as contract commands. Move to tools/generators/ethereum, and make it so we can reference this generator from other packages.
  • To do this, we need to move pkg/chain/gen/cmd static code to a shared library, perhaps under pkg/cmd/ethereum.
  • We also need to adapt the initialize<Contract> flow in generated commands, since it currently leans entirely on how config is handled in keep-core.
  • Lastly, we should consider how we might be able to reuse the Makefile that is currently used. That shouldn't block this issue, as the original directory in keep-core can continue to have a dedicated Makefile.

The convention this introduces is that generators are inside a tools/generators subdirectory, which is rooted in this reasonable-seeming standard.

Adjustments to other repos to use these changes will be tracked on those repos.

Include nonce and gas price parameters for generated mutating contract subcommands

It would be useful for generated commands to have flags for nonce and gas price so that the transactions can be fully customized. Those parameters could be inferred as currently if they were not specified.

Customisable transaction options

Currently, the generated smart contract code does not allow to provide transaction options. Gas price, gas cost and all other parameters are estimated based on the current state of the chain. This is sufficient for simple cases, like smart contract functions with a constant gas cost or smart contract functions called rarely or by just one player.

There are some cases when the estimated gas cost may be wrong. Imagine multiple players trying to submit their lottery tickets at the same time and smart contract code ordering those tickets. In the beginning, there are no tickets and gas cost estimates are based on the execution path with no tickets registered on the chain. If multiple ticket submission transactions are in the mempool, it may happen only the first one gets executed successfully and the rest of them will fail because of out of gas - each transaction after the first one requires some additional sort-related operations to be performed and the gas for these operations was not included in the estimate. To avoid burning their gas, submitters could set a gas limit to the most pessimistic case (but no more!) and have their transactions always executed correctly.

It is now possible to pass Value for payable functions. It seems we don't need From, Nonce or customizable signing function. However, being able to specify GasLimit and GasPrice would be handy.

One possible approach might be to expose TransactionOptions structure and add transactionOptions ...TransactionOptions parameter to each generated transactional function:

type TransactionOptions struct {
	GasLimit uint64
	GasPrice *big.Int
}

type Contract struct {
}

func (c *Contract) SubmitTicket(
	ticketValue *big.Int,
	transactionOptions ...TransactionOptions,
) {
  // (...)
}

and later, to use estimates:

contract.SubmitTicket(
	big.NewInt(1),
)

and to use custom values:

contract.SubmitTicket(
	big.NewInt(1),
	TransactionOptions{
		GasLimit: 1000000,
	},
)