someguy123 / steem-rpc-scanner

Steem RPC scanner in Python using requests-threads and twisted

Hive / Steem-based RPC node scanner

by @someguy123

A fast and easy to use Python script which scans Hive, Steem, and other forks' RPC nodes asynchronously using HTTPX and native Python AsyncIO.

Features:

• Colorized output for easy reading
• Tests a node's reliability during data collection, with multiple retries on error
• Reports the average response time, and average amount of retries needed for basic calls
• Detects a node's Blockchain version
• Show the node's last block number and block time
• Can determine whether a node is using Jussi, or if it's a raw steemd node
• Can scan a list of 20 nodes in as little as 10 seconds thanks to native Python AsyncIO plus the HTTPX AsyncIO requests library

Python 3.8.0 or higher strongly recommended

Python 3.7.x may or may not work

Quickstart with Docker

This RPC scanner is designed to work out-of-the-box with no manual configuration required, other than installing dependencies.

Through the use of Docker of my pre-built docker images, you don't even need to install Python or any library dependencies, it Just Works™

NOTE: As long as you specify the full tag someguy123/rpc-scanner, it will be auto-downloaded from DockerHub if you don't already have the docker image installed.

Docker will not automatically update the image though, you'll need to run the command docker pull someguy123/rpc-scanner occasionally, to update the image when new versions are released.

Quickstart commands

# View the help for app.py and health.py (the main python files containing the various sub-commands)
docker run --rm -it someguy123/rpc-scanner scan --help
docker run --rm -it someguy123/rpc-scanner health --help

# Run the RPC Scanner 'app.py' in quiet mode, which scans all nodes in 'nodes.conf', including individual API tests
# for RPC plugin health, and outputs their health status as a colourful pretty printed table.
# By default, example.nodes.conf is used, unless you attach a custom one as a volume to /app/nodes.conf
docker run --rm -it someguy123/rpc-scanner scan -q --plugins

# Run the RPC scanner 'app.py' and mount /root/my_nodes.conf into the container, so that the scanner uses 
# your custom node list, instead of the default example.nodes.conf
docker run --rm -v /root/my_nodes.conf:/app/nodes.conf -it someguy123/rpc-scanner scan -q --plugins

# Scan an individual RPC node using 'health.py scan' in quiet mode. This emits a standard UNIX exit code,
# with zero (0) if the node gets a score of at least MAX_SCORE - 10 (default: 50 - 10 = 40), and 
# non-zero (default 8) if the node scores less than MAX_SCORE - 10 (default: < 40)
docker run --rm -it someguy123/rpc-scanner health -q scan https://hived.privex.io

# Scan an individual RPC node, but only with API method testing - no preliminary node software identification,
# version retrieval, or stability testing etc. unlike 'app.py' and 'health.py scan'.
# NOTE: There are several helpful aliases available, the below 3 examples all run the same 
# command under-the-hood (./health.py test_methods) 
docker run --rm -it someguy123/rpc-scanner health test_methods https://api.hive.blog
docker run --rm -it someguy123/rpc-scanner test_methods https://api.hive.blog
docker run --rm -it someguy123/rpc-scanner apis https://api.hive.blog
docker run --rm -it someguy123/rpc-scanner meths https://api.hive.blog

# Test only an individual API method. Again, the below 3 commands are equivalent to eachother 
docker run --rm -it someguy123/rpc-scanner health test_method https://hive.3speak.online condenser_api.get_blog
docker run --rm -it someguy123/rpc-scanner meth https://hive.3speak.online condenser_api.get_blog
docker run --rm -it someguy123/rpc-scanner api https://hive.3speak.online condenser_api.get_blog

# To update your rpc-scanner docker image, simply use docker pull
docker pull someguy123/rpc-scanner

Install

Easy way

git clone https://github.com/Someguy123/steem-rpc-scanner.git
cd steem-rpc-scanner

./run.sh install

Manual install (if the easy way isn't working)

# You may need to install the default python version for your distro, for newer python versions
# to work properly (e.g. 'pip' and 'venv' may only be available as python3-pip and python3-venv)
apt install -y python3 python3-dev 
apt install -y python3-pip python3-venv
# Python 3.8+ is recommended, if available on your system.
apt install -y python3.8 python3.8-dev
# If you don't have 3.8 available, python 3.7 may work.
apt install -y python3.7 python3.7-dev

# Install pipenv using the newest version of Python on your system
python3.8 -m pip install -U pipenv

# Clone the repo
git clone https://github.com/Someguy123/steem-rpc-scanner.git
cd steem-rpc-scanner
# Create a virtualenv + install dependencies using pipenv
pipenv install
# Activate the virtualenv
pipenv shell
# Copy the example nodes.conf file into nodes.conf
cp example.nodes.conf nodes.conf

Usage

Scan a list of nodes and output their health info as a colourful table

For most people, the defaults are fine, so you can simply run:

./app.py

Add or delete nodes from nodes.txt line-by-line as needed. You can comment out nodes by placing # at the start of the line.

Format: https://steemd.privex.io - can also specify a port in standard url format, e.g. https://gtg.steem.house:8090

Full usage information (for most up to date usage, use ./app.py --help)

usage: app.py [-h] [-v] [-q] [-f NODEFILE]

Scan RPC nodes from a list of URLs to determine their last block, version,
reliability, and response time.

optional arguments:
  -h, --help   show this help message and exit
  -v           display debugging
  -q           only show warnings or worse
  -f NODEFILE  specify a custom file to read nodes from (default: nodes.txt)

Scan an individual node with UNIX return codes

RPCScanner can easily be integrated with monitoring scripts by using ./health.py scan, which returns a standard UNIX error code based on whether that RPC is working properly or not.

Example 1 - Scanning fully functioning RPC node

user@host ~/rpcscanner $ ./run.sh health -q scan "https://hived.privex.io/"

Node: http://hived.privex.io/
Status: PERFECT
Network: Hive
Version: 0.23.0
Block: 43810613
Time: 2020-05-29T00:30:24 (0:00:00 ago)
Plugins: 8 / 8
PluginList: ['condenser_api.get_followers', 'bridge.get_trending_topics', 'condenser_api.get_accounts', 'condenser_api.get_witness_by_account', 'condenser_api.get_blog', 'condenser_api.get_content', 'condenser_api.get_account_history', 'account_history_api.get_account_history']
PassedStages: 3 / 3
Retries: 0
Score: 50 (out of 50)

user@host ~/rpcscanner $ echo $?
0

As you can see, hived.privex.io got a perfect score of 20, and thus it signalled the UNIX return code 0, which means "everything was okay".

Example 2 - Scanning a misbehaving RPC node

user@host ~/rpcscanner $ ./run.sh health -q scan "https://steemd.privex.io/"

Node: http://steemd.privex.io/
Status: BAD
Network: Steem
Version: error
Block: 43536277
Time: 2020-05-20T13:59:57 (8 days, 10:31:40 ago)
Plugins: 4 / 8
PluginList: ['condenser_api.get_account_history', 'condenser_api.get_witness_by_account', 'condenser_api.get_accounts', 'account_history_api.get_account_history']
PassedStages: 2 / 3
Retries: 0
Score: 2 (out of 50)

user@host ~/rpcscanner $ echo $?
8

Unfortunately, steemd.privex.io didn't do anywhere near as well as hived.privex.io - it scored a rather low 7 / 20, with only 4 of the 8 RPC calls working properly which were tested.

This resulted in health.py signalling return code 8 instead (non-zero), which tells a calling program / script that something went wrong during execution of this script.

In this case, 8 is the default setting for BAD_RETURN_CODE, giving a clear signal to the caller that it's trying to tell it "the passed RPC node's score is below the threshold and you should stop using it!".

You can change the numeric return code used for both "good" and "bad" results from the individual node scanner by setting GOOD_RETURN_CODE and/or BAD_RETURN_CODE respectively in .env:

# There isn't much reason to change GOOD_RETURN_CODE from the default of 0. But the option is there if you want it.
GOOD_RETURN_CODE=0
# We can change BAD_RETURN_CODE from the default of 8, to 99 for example.
# Any integer value from 0 to 254 can generally be used.
BAD_RETURN_CODE=99

Making use of these return codes in an external script

Included in the extras folder of the repo, are two example scripts - one in plain old Bash (the default terminal shell of most Linux distro's and macOS), and a python script, intended for use on Python 3.

Both scripts do effectively the same thing - they load nodes.txt, skipping any commented out nodes, then check whether each one is fully functional or not by calling health.py scan NODE, and check for a non-zero return code. Then outputting either a green UP NODE http://example.com or a red DOWN NODE http://example.com.

Pictured above is a screenshot of both the bash example, and the python example - running with the same node list, and same version of this RPC Scanner.

Handling program return codes is generally going to be the easiest in shell scripting languages, including Bash - as most shell scripting languages are built around the UNIX methodology - everything is a file, language syntax is really just executing programs with arguments, and return codes from those programs power the logic syntax etc.

The most basic shell script would be a simple if call, using /path/to/health.py scan http://somenode as the if test. Most shells such as Bash will read the return (exit) code of the program, treating 0 as "true" and everything else as "false".

Basic shell script example

#!/usr/bin/env bash

if /opt/rpcscanner/health.py scan "https://hived.privex.io" &> /dev/null; then
    echo "hived.privex.io is UP :)"
else
    echo "hived.privex.io is DOWN!!!"
fi

License

GNU AGPL 3.0

See file LICENSE

Common environment settings

• RPC_TIMEOUT (default: 3) Amount of seconds to wait for a response from an RPC node before giving up.

• MAX_TRIES (default: 3) Maximum number of attempts to run each call against an RPC node. Note that this number includes the initial try - meaning that setting MAX_TRIES=1 will disable automatic retries for RPC calls.

  DO NOT set this to 0 or the scanner will simply think all nodes are broken. Setting MAX_TRIES=0 may however be useful if you need to simulate how an external application handles "DEAD" results from the scanner.

• RETRY_DELAY (default: 2.0) Number of seconds to wait between retrying failed RPC calls. Can be a decimal number of seconds, e.g. 0.15 would result in a 150ms retry delay.

• PUB_PREFIX (default: STM) The first 3 characters at the start of a public key on the network(s) you're testing. This is used by rpcscanner.MethodTests.MethodTests for thorough "plugin tests" which validate that an account's public keys look correct.

• GOOD_RETURN_CODE (default: 0) The integer exit code returned by certain parts of RPCScanner, e.g. health.py scan [node] when the given RPC node(s) are functioning fully.

• BAD_RETURN_CODE (default: 0) The integer exit code returned by certain parts of RPCScanner, e.g. health.py scan [node] when the given RPC node(s) are severely unstable or missing vital plugins.

steem-rpc-scanner's Issues

Nodes show up as dead but are working

Some examples:

• https://api.steem.house/
  https://api.steem.house DEAD error error error error error

• https://rpc.buildteam.io
  https://rpc.buildteam.io DEAD error error error error error

Seems about half of working servers show up as DEAD even though they are alive.

Suggestion: Sort results

I think it would be easier to read if the results were sorted with working nodes up top and dead nodes on bottom.