add fail-safe mode by default to avoid double-sign when the new pod started while the old one is terminating but is still up about cosmos-omnibus

[Dimokus88]

i found value double_sign_check_height = 0 in config.toml . But i not sure about reliability.
But this parameter will look good as an addition to the script.

[andy108369]

I remember I was using this script for Crypto Org chain for checking the last N signatures.
So its parts can be leveraged for the fail-safe check.

#!/bin/bash
# filename: ./scr
address=$1
hist=$2

if [ -z "$hist" ] || [ -z $address ] || [ ! $hist -gt 0 ] ; then
    echo ""
    echo -e "Usage:\n\t$0 validator_address number_of_history_blocks"
    echo ""
    echo -e "Example:\n\t$0 0E0DA7A5005429525A080C026D8DB0E7C85C5A11 20"
    echo ""
    echo "Hint: you can find your validator with the command:"
    echo -e "\tjq -r '.address' ~/.chain-maind/config/priv_validator_key.json"
    echo ""
    exit
fi

cheight=$(curl -s http://127.0.0.1:26657/commit | jq -r .result.signed_header.header.height)
lheight=$(expr $cheight - $hist)

sigs=0
for (( height=$lheight ; height < $cheight ; height++)); do
    JSON=$( curl -s http://127.0.0.1:26657/block?height=$height )
    sig=$( echo $JSON | jq -r .result.block.last_commit.signatures | grep -c $address )
    TISO=$( echo $JSON | jq .result.block.header.time | sed 's/\..*Z/Z/' | jq '( fromdateiso8601 )')
    TS=$( date -d "@$TISO" +'%Y-%m-%d %H:%M:%S' )
    echo -e "Local time: $TS\tBlock#: $height\tSigned: $sig"
    sigs=$(( $sigs + $sig ))
done

perc=$( echo "$sigs / ( $hist / 100 )" | bc -l )
missed=$(( $hist - $sigs ))

printf '\nValidator %s signed %.2f%s of the last %d blocks.\n' $address $perc '%' $hist
echo "Number of missed signatures: ${missed}."

Check if validator is signing the blocks:

$ jq -r '.address' ~/.chain-maind/config/priv_validator_key.json
666A630D5E13294072F2D1ECA7655CAA4F84289F

$ ./scr 666A630D5E13294072F2D1ECA7655CAA4F84289F 5
Local time: 2021-03-26 16:44:53 Block#: 22055   Signed: 1
Local time: 2021-03-26 16:45:00 Block#: 22056   Signed: 1
Local time: 2021-03-26 16:45:06 Block#: 22057   Signed: 1
Local time: 2021-03-26 16:45:13 Block#: 22058   Signed: 1
Local time: 2021-03-26 16:45:19 Block#: 22059   Signed: 1

Validator 666A630D5E13294072F2D1ECA7655CAA4F84289F signed 100.00% of the last 5 blocks.
Number of missed signatures: 0.

[Dimokus88]

I remember I was using this script for Crypto Org chain for checking the last N signatures. So its parts can be leveraged for the fail-safe check.

#!/bin/bash
# filename: ./scr
address=$1
hist=$2

if [ -z "$hist" ] || [ -z $address ] || [ ! $hist -gt 0 ] ; then
    echo ""
    echo -e "Usage:\n\t$0 validator_address number_of_history_blocks"
    echo ""
    echo -e "Example:\n\t$0 0E0DA7A5005429525A080C026D8DB0E7C85C5A11 20"
    echo ""
    echo "Hint: you can find your validator with the command:"
    echo -e "\tjq -r '.address' ~/.chain-maind/config/priv_validator_key.json"
    echo ""
    exit
fi

cheight=$(curl -s http://127.0.0.1:26657/commit | jq -r .result.signed_header.header.height)
lheight=$(expr $cheight - $hist)

sigs=0
for (( height=$lheight ; height < $cheight ; height++)); do
    JSON=$( curl -s http://127.0.0.1:26657/block?height=$height )
    sig=$( echo $JSON | jq -r .result.block.last_commit.signatures | grep -c $address )
    TISO=$( echo $JSON | jq .result.block.header.time | sed 's/\..*Z/Z/' | jq '( fromdateiso8601 )')
    TS=$( date -d "@$TISO" +'%Y-%m-%d %H:%M:%S' )
    echo -e "Local time: $TS\tBlock#: $height\tSigned: $sig"
    sigs=$(( $sigs + $sig ))
done

perc=$( echo "$sigs / ( $hist / 100 )" | bc -l )
missed=$(( $hist - $sigs ))

printf '\nValidator %s signed %.2f%s of the last %d blocks.\n' $address $perc '%' $hist
echo "Number of missed signatures: ${missed}."

Check if validator is signing the blocks:

$ jq -r '.address' ~/.chain-maind/config/priv_validator_key.json
666A630D5E13294072F2D1ECA7655CAA4F84289F

$ ./scr 666A630D5E13294072F2D1ECA7655CAA4F84289F 5
Local time: 2021-03-26 16:44:53 Block#: 22055   Signed: 1
Local time: 2021-03-26 16:45:00 Block#: 22056   Signed: 1
Local time: 2021-03-26 16:45:06 Block#: 22057   Signed: 1
Local time: 2021-03-26 16:45:13 Block#: 22058   Signed: 1
Local time: 2021-03-26 16:45:19 Block#: 22059   Signed: 1

Validator 666A630D5E13294072F2D1ECA7655CAA4F84289F signed 100.00% of the last 5 blocks.
Number of missed signatures: 0.

Looks a awesome!
I think, this can implementation at running service in deployment!

[andy108369]

The only caveat is that one would have to use some known RPC node to query as one can't just query itself 127.0.0.1:26657 before starting the node...

I am wondering, maybe tendermint already has this kind of check built-in?

[andy108369]

i found value double_sign_check_height = 0 in config.toml . But i not sure about reliability. But this parameter will look good as an addition to the script.

yeah, it looks like exactly that! thank you!

https://docs.tendermint.com/v0.34/tendermint-core/configuration.html

[consensus]
# ...

# How many blocks to look back to check existence of the node's consensus votes before joining consensus
# When non-zero, the node will panic upon restart
# if the same consensus key was used to sign {double_sign_check_height} last blocks.
# So, validators should stop the state machine, wait for some blocks, and then restart the state machine to avoid panic.
double_sign_check_height = 0

[Dimokus88]

For nodes on Akash , this is an important parameter, as validators run the risk of being permanently banned.
And double containers at the moment are quite common among providers.

[andy108369]

For nodes on Akash , this is an important parameter, as validators run the risk of being permanently banned. And double containers at the moment are quite common among providers.

I don't think the ban is permanent. Validator can always issue the tx slashing unjail to get out of the jail.

[Dimokus88]

For nodes on Akash , this is an important parameter, as validators run the risk of being permanently banned. And double containers at the moment are quite common among providers.

I don't think the ban is permanent. Validator can always issue the tx slashing unjail to get out of the jail.

A double signature is punishable by an eternal ban, it looks like the date unjail 9999-12-31

[andy108369]

For some reason some had a problem with that option (double_sign_check_height) enabled.
So I think it needs to be tested.

https://discord.com/channels/747885925232672829/751246770885099533/968578447884550146

[andy108369]

So I think it needs to be tested.

I've just tested it with SIFNODED_CONSENSUS_DOUBLE_SIGN_CHECK_HEIGHT=10, it worked very well:

11:37PM INF found signature from the same key height=9303106 idx=41 module=consensus sig={"block_id_flag":2,"signature":"REDACTED","timestamp":"2022-11-05T23:35:43.456919786Z","validator_address":"REDACTED"}
panic: Failed to start consensus state: found signature from the same key