nearprotocol / stakewars

Stake Wars leaderboard and place to report issues

Stake Wars Episode II: Return of the Validators

September 25rd 2020 update: contribution opportunities will be over next week.

Welcome to NEAR Stake Wars Episode II, Return of the Validators!

This page will guide you through three main aspects of this initiative:

1. In a Few Words
2. What is NEAR Stake Wars
3. Your toolbox
4. What you have to do

💁 In a Few Words

Stake Wars is the program that accelerates your path to become a validator on NEAR Protocol. It is structured in technical challenges of increasing difficulty, giving you the opportunity of "learning by doing". Validators that will complete all the challenges will prove to be ready to join MainNet Restricted and will be recognized as early supporters of NEAR Protocol.

Quickstart

If you want to get your feet wet, and see if Stake Wars is for you, try this quickstart guide:

• Create a BetaNet account using our hosted wallet here.
• Spin up your Ubuntu/Debian VPS and run nearup.
• Deploy your staking pool using the Staking Pool Factory
• Submit up this form to enroll in the Stake Wars and receive your BetaNet tokens.

Stake Wars received an exceptional interest by the community of validators (500+ applications, 300+ PRs). We are processing all of them, sending 75,000 BetaNet tokens as promised, while making sure the network runs with no issues. As explained in the docs, there are 100 seats per shard, so BetaNet is running at capacity with a cost per seat that could be above the 75.000 tokens we normally provide.

You can receive more BetaNet tokens by completing the available challenges and submitting contributions on the NEAR community portal.

Here is the list of the available challenges:

• 001: Create your BetaNet wallet, deploy your node, and correctly configure your staking pool.
• 002: Enroll your staking pool, receive your delegation, and maintain your validator status!
• 003: Monitor your node health, and send an automated email in case of issues.
• 004: Create a warchest of staked tokens, and dynamically maintain no more than one validator seat.
• 005: Automatically deploy nearcore using a CI/CD pipeline.

After you successfully complete the challenges, and you maintain a high uptime of your node, you will be invited by NEAR team to run your node on TestNet, and become a candidate run MainNet Restricted.

🚀 What is NEAR Stake Wars

Stake Wars is NEAR's incentivized testnet for professional validators.

NEAR’s MainNet recently launched into its first phase, called “POA” (see full roadmap). This means that a small handful of validating nodes are currently being run by the core team. In order to progress to the next phase, “MainNet: Restricted”, the operation of the network will be handed off to a large group of node operators called validators.

The goal of Stake Wars: Episode II is to onboard those validators, test the stability of their system, and begin introducing some of the unique aspects of NEAR’s delegation in preparation for the next phase of MainNet itself.

If you want to know more, read the Stake Wars Episode II blog post.

Blogpost TL;DR:

Stake Wars Ep.II introduces NEAR's contract-based delegation, offering validators the opportunity to take part in the Open Finance ecosystem. There is a staking pool reference contract on Github, ready for experimenting with these principles. Deploying the staking pool and participating in the Stake Wars will give access to NEAR's MainNet Restricted. Rewards will include 10,000 NEAR tokens/month for every validator on MainNet Restricted, plus 1 Million NEAR tokens available for contributions and community challenges. To become validators on MainNet Restricted, participants will have to accomplish technical challenges and successfully run nodes on BetaNet and TestNet. Judgment criteria will be quantitative, such as number of blocks generated and uptime; and qualitative, such as reactivity to network updates and community participation. A Validator Advisory Board, with a selected group of professional validators, will become over time the voice of validators in the technical governance of the protocol

🔧 Your Toolbox

NEAR Protocol provides you multiple tools such as Github repositories, applications, documentation and web-based resources. As a Stake Wars participant you will need all of them.

Github Repositories

NEAR is using two main accounts: github.com/nearprotocol and github.com/near. To join Stake Wars you will use:

• nearup, public scripts to launch NEAR Protocol devnet, betanet and testnet nodes
• nearcore, the reference client for NEAR Protocol
• near-cli, the general purpose command line tools for interacting with NEAR Protocol
• core-contracts, where you can find the reference staking pool smart contract

NEAR Documentation

Most of the technical documentation is available at docs.near.org. An entire section is dedicated to validators.

NEAR Online Resources

The website provides a block explorer and a web wallet:

• BetaNet web wallet: wallet.betanet.near.org
• BetaNet explorer: explorer.betanet.near.org

You will need a BetaNet wallet to deploy your staking pool and receive your first delegation.

You can also use the JSON RPC interface at the address https://rpc.betanet.near.org/status to quickly retrieve information about network, blocks, transactions and wallets. There's also a documentation section with an overview of the available RPC endpoints.

As a final note, https://status.nearprotocol.com/ will give you feedback on the status of the network, and the most recent incidents.

NEAR Community channels

Connect to other validators using the dedicated channel on Discord. You might join also NEAR Validators on Telegram.

Important: NEAR core team will use the prefix [CODE_RED] if a particular message requires technical attention by validators. Some examples are new releases, hard forks and critical issues.

Overall, if you want to successfully participate in the Stake Wars, you'll have to:

• Keep an eye to Validator Announcements on Discord (there's a dedicated channel). It will be used to inform you about technical releases and hard forks, community challenges, contribution opportunities and other initiatives that will be valuable for you.
• Give and receive technical help here on Github, in the issues section.

Other files is in this repo

• GUIDELINES.md: some basic rules to follow, and what to do when you are invited to join TestNet
• updates.md: good to catch up if you missed the last few weeks
• VALIDATORS.md: who else is participating in the Stake Wars
• LEADERBOARD.md: how validators are doing in terms of uptime
• troubleshooting.md: a colletion of the most common issues
• staking_cheatsheet.md: a list of the most common commands to manage the staking pool
• scenarios.md: some high-level examples of what you will have to do

...and the challenges folder...

🏆 What You Have to Do

As you know, validators are responsible to generate new blocks to update the state of the network. NEAR Protocol uses proof-of-stake to secure its infrastruture, so you have to stake tokens to become a validator. The contract-based delegation, as explained in the Stake Wars Ep.II blog post, will provide you stake from users who don't want to run a node but are interested to secure the network - and earn rewards with you.

If your mission is to become a validator on MainNet, you have two ways to succeed:

1. Wait until NEAR enters Phase Two of MainNet (see the roadmap here)
2. Complete the Stake Wars challenges, be invited to join TestNet, and prove that you are ready for the Phase One, called MainNet Restricted

Validators joining MainNet Restricted will receive help from NEAR foundation in the form of a grant in NEAR tokens, and tokens delegation to help them retain their validator seat while they onboard their customers. The plan is to onboard ~50 validators in the Phase One, so if you are interested you simply have to successfully complete the challenges below!

List of Validator Challenges

We will publish new challenges for validators on a regular basis. Every challenge will have a progressive number, with an increasing level of difficulty. The acceptance criteria will provide high-level indications, and some of these challenges may list previous challenges as a requirement.

List of challenges:

1. May 25th 2020, Challenge 001 Deploy your node and your staking pool
2. May 25th 2020, Challenge 002 Become a validator and manage your seat
3. June 8th 2020, Challenge 003 Monitor your node and setup automated alerts
4. June 22nd 2020, Challenge 004 Dynamically adjust your stake
5. July 17 2020, Challenge 005 Automatically deploy nearcore using a CI/CD pipeline.

stakewars's Issues

[TEST][ Dangerous ] unpredictable errors may be caused by Cross platform compilation.

Running near program compiled in a single CPU platform on DOCKER of different CPU platforms may lead to unpredictable errors!

These days, I have been looking for the reason of the errors mentioned in issues #25 and finally I found that the problem lies in the incompatible CPU platform instruction set .
As mentioned in issues #25, there will be illegal instruction errors when I runs the nearcore with docker. Now I can confirm that It was caused by incompatible instruction sets between my CPU platform and the platform on where near was compiled.

In order to prove my idea, I mapped the locally compiled near program into the container of nearcore and tried to run it. It turned out that everything was OK. I was right.

map local near program into container:

root@file:~# docker run -ti -v /root/.near:/srv/near -v ~/nearcore/target/release:/testnear 6537accf533c /bin/bash

local folder ~/nearcore/target/release mapped into container folder /testnear

root@247329f6a649:/# cd testnear
root@247329f6a649:/testnear# ls
build deps examples genesis-csv-to-json genesis-csv-to-json.d incremental libnear.d libnear.rlib native near near.d

error occured when run near in the PATH of container OS

root@247329f6a649:/testnear# near --home /srv/near run
[2019-11-07T12:10:09Z INFO near] Opening store database at "/srv/near/data" Illegal instruction (core dumped)

node startup normally when run near in the folder /testnear which was compiled locally.

root@247329f6a649:/testnear# ./near --home /srv/near run
[2019-11-07T12:10:15Z INFO near] Opening store database at "/srv/near/data"
[2019-11-07T12:10:15Z INFO near_network::peer_manager] Server listening at ed25519:[email protected]:24567
[2019-11-07T12:10:25Z INFO near_client::info] # 0 Waiting for peers V/38 1/1/40 peers ⬇ 2 B/s ⬆ 2 B/s 0.00 bps 0 gas/s CPU: 7%, Mem: 22.3 MiB
[2019-11-07T12:10:29Z WARN near_network::peer] Received Sync while Connecting from Outbound connection.
[2019-11-07T12:10:35Z INFO near_client::info] State 2: header V/38 3/1/40 peers ⬇ 8.4kiB/s ⬆ 8.3kiB/s 0.00 bps 0 gas/s CPU: 5%, Mem: 24.8 MiB

Mention the branch for Stakewars

Describe the bug or test
I want to validate, but always before validating, I check if there are updates on the branch. Can we update the readme on which branch the validators have to be on.

Expected behavior OR Outcome of test
Find in the README what branch we should be on for StakeWars as the docs of the offical website mention staging.

Machine Specs:

Cloud: TencentCloud
CPU: 2 core
RAM: 4Gb
Bandwidth：5Mbps
Disk: 50GB SSD

abnormal exit while waiting peer

[2019-10-22T10:54:16Z INFO near_client::info] # 28780 Waiting for peers -/1 2/1/40 peers ⬇ 1.9MiB/s ⬆ 20.4kiB/s 0.00 bps 0 gas/s CPU: 36%, Mem: 134.0 MiB
[2019-10-22T10:54:26Z INFO near_client::info] # 28780 Waiting for peers -/1 2/1/40 peers ⬇ 1.9MiB/s ⬆ 20.5kiB/s 0.00 bps 0 gas/s CPU: 43%, Mem: 134.0 MiB
[2019-10-22T10:54:36Z INFO near_client::info] # 28780 Waiting for peers -/1 2/1/40 peers ⬇ 1.9MiB/s ⬆ 20.8kiB/s 0.00 bps 0 gas/s CPU: 29%, Mem: 134.0 MiB
[2019-10-22T10:54:46Z INFO near_client::info] # 28780 Waiting for peers -/1 2/1/40 peers ⬇ 1.8MiB/s ⬆ 19.4kiB/s 0.00 bps 0 gas/s CPU: 40%, Mem: 134.0 MiB
[2019-10-22T10:54:56Z INFO near_client::info] # 28780 Waiting for peers -/1 2/1/40 peers ⬇ 1.8MiB/s ⬆ 19.5kiB/s 0.00 bps 0 gas/s CPU: 39%, Mem: 134.0 MiB
[2019-10-22T10:55:06Z INFO near_client::info] # 28780 Waiting for peers -/1 2/1/40 peers ⬇ 1.8MiB/s ⬆ 19.2kiB/s 0.00 bps 0 gas/s CPU: 45%, Mem: 134.0 MiB
[2019-10-22T10:55:16Z INFO near_client::info] # 28780 Waiting for peers -/1 2/1/40 peers ⬇ 1.9MiB/s ⬆ 20.5kiB/s 0.00 bps 0 gas/s CPU: 53%, Mem: 134.0 MiB
[2019-10-22T10:55:26Z INFO near_client::info] # 28780 Waiting for peers -/1 2/1/40 peers ⬇ 2.0MiB/s ⬆ 22.2kiB/s 0.00 bps 0 gas/s CPU: 46%, Mem: 134.0 MiB
/usr/local/bin/run.sh: line 24: 25 Illegal instruction (core dumped) near --home=${NEAR_HOME} run --boot-nodes=${BOOT_NODES}
*** /usr/local/bin/run.sh exited with status 132.
*** Shutting down runit daemon (PID 20)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
Oct 22 10:55:29 0c7397ffd925 syslog-ng[13]: syslog-ng shutting down; version='3.13.2'
*** Killing all processes...

Add Prompt in start_stakewars.py script to prevent key file overwritten by mistake

Describe the bug or test
When you exec ./scripts/start_stakewars.py with --init options, it always gen new node_key.json and validator_key.json even if old ones exist. But sometimes, it is just a misoperation such as miscopy, repeat cmd history by accident, and so on. The operator does NOT want to have his key files overwritten.
So, I suggest to add some check mechanism, if see files with same name in the dest dir, just give a prompt and let operator to choose going on to overwrite them or just stop to make a backup.

How to Reproduce
Steps to reproduce the behavior:
as mentioned above.

Expected behavior OR Outcome of test
as mentioned above.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AliYun
• CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
• RAM: 4G
• Disk space: 500G SSD

Additional context
as mentioned above.

Screenshots Optional
N/A

[Choose one --> BUG] wallet GUI still shows account from previous stakewars network

Describe the bug or test
Attached screen capture.

as shown in the screen capture, only vow5 was created in current network.
the rest of the accounts still showing in the account dropdown box for user to switch to
likes
validators_online, jleonyw4, jleonyw4b, jleony

How to Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See thing

Expected behavior OR Outcome of test
A clear and concise description of what you expected to happen.

Specs (please complete the following information):

• OS:
• Which cloud / on-premise:
• CPU:
• RAM:
• Disk space:

Additional context
Add any other context about the problem here.

Screenshots Optional
If applicable, add screenshots to help explain your problem.

[TEST][bug] (Resolved)near launch error when run with docker

Describe the bug or test
start node by run: root@file:~/nearcore# ./scripts/start_stakewars.py
an error occured:
[2019-11-05T14:25:16Z INFO near] Opening store database at "/srv/near/data" /usr/local/bin/run.sh: line 24: 24 Illegal instruction (core dumped) near --home=${NEAR_HOME} run --boot-nodes=${BOOT_NODES}

*** /usr/local/bin/run.sh exited with status 132.
*** Shutting down runit daemon (PID 19)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
Nov 5 14:25:17 e054486da90c syslog-ng[12]: syslog-ng shutting down; version='3.13.2' *** Killing all processes...

To find what cause the error,I enter the docker container :

root@file:~/nearcore# docker run -ti -v /root/.near:/srv/near -v /tmp:/tmp -p 24567:24567 -p 3030:3030 6537accf533c /bin/bash

In the container , I run the command:root@62286e684a59:/# near --home /srv/near run
The same error " Illegal instruction "

[2019-11-05T14:22:20Z INFO near] Opening store database at "/srv/near/data" Illegal instruction (core dumped)

Specs (please complete the following information):

• OS:Ubuntu 18.04.1 LTS bionic
• CPU: XEON E7
• RAM: 32G
• Disk space:2T

[TEST][bug] Issue generating node, validator, signer key (possible AWS issue)

Describe the bug or test
Trying to run sudo ./scripts/start_stakewars.py --init --account-id=<your_account_id> to generate the 3 needed keypairs

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://github.com/nearprotocol/stakewars' and follow the guide
2. Reach the initialization step, run

sudo ./scripts/start_stakewars.py --init --account-id=<your_account_id>

or

sudo ./scripts/start_stakewars.py --init --signer-keys --account-id=<your_account_id>

3. View contents of file in ~/.near/

4. Files are all empty!

Expected behavior OR Outcome of test
Files should have keys in them... something is going on in https://github.com/nearprotocol/nearcore/blob/master/scripts/nodelib.py

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AWS (@bowenwang1996 mentioned this may be an AWS specific issue)
• CPU: 8
• RAM: 32GB
• Disk space: 100GB

Additional context
Docker is running

LMK if you want to log into my node to test

[BUG] Error on receival of header: DB Not Found Error

Describe the bug or test
The log of nearcore outputs like below a lot:

[2019-11-28T03:03:33Z ERROR near_client::client_actor] Error on receival of header: DB Not Found Error: 6jqcMy3CYz8HGRe111oR5biA3BnCsmuadHVbJjRPLAUp
     Cause: Unknown
     Backtrace:

How to Reproduce
Steps to reproduce the behavior:
It didn't appear for last weeks, but show an error of this week's testnet.

1. Follow the doc To get the local node started
2. After downloading all headers, this error message will output

Expected behavior OR Outcome of test
It should not output the Cause: Unknow or Backtrace: things.

Specs (please complete the following information):

• OS: Ubuntu 16.04.3 LTS
• Which cloud / on-premise: Ucloud
• CPU: 2core
• RAM: 4G
• Disk space: 200G SSD

Screenshots Optional

[TEST][benchmark] Docker setup only takes advantage of 1 cpu

100% usage on cpu core 1 and 0% on cpu core 2
To Reproduce
Steps to reproduce the behavior:

1. installed validator
2. let it run as part of stake wars

Expected behavior OR Outcome of test

Specs (please complete the following information):

• OS:ubuntu 18.04
• Which cloud / on-premise: aws
• CPU: 2
• RAM: 4
• Disk space: 50

[ BUG] Mnemonic word verification failed on Tatooine website

Describe the bug or test
Failed to create an account on https://wallet.tatooine.nearprotocol.com,
after the Mnemonic words are displayed on the website, you are always failed to verify these mnemonic words.

How to Reproduce
Steps to reproduce the behavior:

1. Go to https://wallet.tatooine.nearprotocol.com,
2. Click on create new account
3. Go to verify the mnemonic words
4. See the failed verification

Expected behavior OR Outcome of test
Should verify the mnemonic words sucessfully.

Specs (please complete the following information):

• OS: Winddos10
• Which cloud / on-premise: Local
• CPU:
• RAM:
• Disk space:

Additional context
Add any other context about the problem here.

Screenshots Optional
If applicable, add screenshots to help explain your problem.

[BREAK - P1][epic] near-shell crashed on Ubuntu

Describe the epic break you found or epic contract you've created
near-shell version 0.14.0 crashed when trying to new_project with an Error:
Error: Cannot find module '/usr/local/bin/cli.js'
To Reproduce
Steps to reproduce the behavior:

1. npm -g install npm
2. npm -g install near-shell
3. near new_project stakewars
4. See : Error: Cannot find module '/usr/local/bin/cli.js'

Expected behavior OR Outcome of work
Clearly it should create a dir as it does in version 0.11.x

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AliYun
• CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
• RAM: 4G
• Disk space: 500G SSD

Additional context
npm version: 6.13.0
near-shell version: 0.14.0
And I found this cli.js in this path:
/usr/local/lib/node_modules/near-shell/bin/cli.js
And If I run near with full origin path (not the linker one in /usr/local/bin), it seems OK:

root@buildlinks-test-platform:~/near_stakewars# /usr/local/lib/node_modules/near-shell/bin/near  new_project stakewars
[WARNING] Didn't find config at /root/near_stakewars/src/config, using default shell config
{
  networkId: 'default',
  nodeUrl: 'https://rpc.nearprotocol.com',
  contractName: 'near-hello-devnet',
  walletUrl: 'https://wallet.nearprotocol.com'
}

Screenshots Optional

root@buildlinks-test-platform:~/near_stakewars# near new_project stakewars
internal/modules/cjs/loader.js:775
    throw err;
    ^

Error: Cannot find module '/usr/local/bin/cli.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:772:15)
    at Function.Module._load (internal/modules/cjs/loader.js:677:27)
    at Function.Module.runMain (internal/modules/cjs/loader.js:999:10)
    at internal/main/run_main_module.js:17:11 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

near-shell can use postfix to simplify big number param

Describe the bug or test
In near stake cmd,
near stake <account_id> <PK> <staking amount>
we should put a big number for staking amount, usually include 18 zeros.
I strongly suggest we can use 30k for 30,000, 30m for 30,000,000, 30g for 30,000m, and so on.
Then If we want to stake 10 Near, I can put 10e for 10 followed by 18 zeros.
Here is the list:

• 1KB = 1024 B, K for 3 zero
• 1 MB = 1024KB, M for 6 zero
• 1 GB = 1024 MB, G for 9 zero
• 1 TB = 1024 GB, T for 12 zero
• 1 PB = 1024 TB, P for 15 zero
• 1 EB = 1024 PB, E for 18 zero

How to Reproduce
This is a suggestion.

Expected behavior OR Outcome of test
As described in first section.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AliYun
• CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
• RAM: 4G
• Disk space: 500G SSD

Additional context
N/A

Screenshots Optional
N/A

NODE CRASH caused by a sudden memory and cpu soar

Describe the bug or test
At height 34905, usage of mem and cpu both soar, cpu from 4% to 100%, mem from 0.39G to 4G and cause a oomkiller process. After auto restart, node then halt at heigh 34914, and say: Received Sync while Connecting from Outbound connection.
And finally, node stuck at 35002, and says Downloading headers 100%.

How to Reproduce
Steps to reproduce the behavior:

1. Happened at 2019-11-13T06:35:29
2. see screen shot for detail info

Expected behavior OR Outcome of test
running smoothly.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AliYun
• CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
• RAM: 4G
• Disk space: 500G SSD

Additional context

Screenshots Optional
first shot:

after auto restart:

at some point of restarting, it shows different:

finally:

Create Account Error in online wallet

Describe the bug or test
When create a new account, first it says ok, your id is available, but when click Create, after a while, it says "Username is taken. Try something else.". And some error msg showed in validator node.

How to Reproduce
Steps to reproduce the behavior:

1. Go to online wallet
2. Add some account
3. click Create
4. wait a while, it will show "Username is taken. Try something else."

Expected behavior OR Outcome of test
Clearly, the name is not occupied, the creation should success.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AliYun
• CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
• RAM: 4G
• Disk space: 500G SSD

Additional context
Screenshots Optional

[BUG] Stakewars node shows in Testnet explorer

Describe the bug or test
My Stakewars validator node (stakedincweek2) shows as a node in the testnet explorer and not in the tatooine explorer as I would expect. My node's block number is consistent with the tatooine's current block, not testnet.

How to Reproduce
Steps to reproduce the behavior:

1. No config.json, genesis.json, or data/ in NEAR home
2. start with scripts/start_stakewars.py --home <NEAR home> (Docker run)
   a. be a validator (not sure if this is important)
3. Wait for sync
4. Check list of Testnet and Tatooine nodes

Expected behavior OR Outcome of test
Tatooine explorer displays the correct set of nodes.

Specs (please complete the following information):

• OS: Amazon Linux 2
• Which cloud / on-premise: Cloud (AWS EC2)
• CPU: 8
• RAM: 32GB
• Disk space: 500GB

Additional context
Tatooine explorer may be displaying nodes prior to the Stakewars reset on 11/18 at 6:23p EST and Testnet is displaying nodes after that reset.

nearprotocol/nearcore Docker image digests tested:

sha256:2c788d4ec7b67469cc30303f1535ad700833033c42def2b3efeb2f91002271b1
sha256:4cef8d30b3713c669661cd23541ee8a19f7fda8b434e0a965cfcf7b3cdbde4ac

config.json

{
  "genesis_file": "genesis.json",
  "validator_key_file": "validator_key.json",
  "node_key_file": "node_key.json",
  "rpc": {
    "addr": "0.0.0.0:3030",
    "cors_allowed_origins": [
      "*"
    ],
    "polling_config": {
      "polling_interval": {
        "secs": 0,
        "nanos": 500000000
      },
      "polling_timeout": {
        "secs": 10,
        "nanos": 0
      }
    }
  },
  "telemetry": {
    "endpoints": [
      "https://explorer.nearprotocol.com/api/nodes"
    ]
  },
  "network": {
    "addr": "0.0.0.0:24567",
    "external_address": "",
    "boot_nodes": "ed25519:[email protected]:24567,ed25519:[email protected]:24567,ed25519:[email protected]:24567,ed25519:[email protected]:24567,ed25519:[email protected]:24567,ed25519:[email protected]:24567",
    "max_peers": 40,
    "handshake_timeout": {
      "secs": 20,
      "nanos": 0
    },
    "reconnect_delay": {
      "secs": 60,
      "nanos": 0
    },
    "skip_sync_wait": false,
    "ban_window": {
      "secs": 10800,
      "nanos": 0
    }
  },
  "consensus": {
    "min_num_peers": 3,
    "block_production_tracking_delay": {
      "secs": 0,
      "nanos": 100000000
    },
    "min_block_production_delay": {
      "secs": 1,
      "nanos": 0
    },
    "max_block_production_delay": {
      "secs": 2,
      "nanos": 0
    },
    "max_block_wait_delay": {
      "secs": 6,
      "nanos": 0
    },
    "reduce_wait_for_missing_block": {
      "secs": 0,
      "nanos": 100000000
    },
    "produce_empty_blocks": true
  },
  "tracked_accounts": [],
  "tracked_shards": []
}

[TEST] Hard to sync new block due to many invalid block Header and DB Not Found Error

Describe the bug or test
When some nodes are running with old genesis file and old code, others are running with new genesis and new code, log will show many info about receive invalid block Header and .Error processing sync blocks due to DB Not Found Error.

Although it seems correct according to chain runtime logic, but it consumes nodes resource, let node confused about peer which should or should not to connect, and stucked at some height waiting downloading some block header.

My suggestion is,

• using some kind of version number to distinguish different code, prevent node with different code version from establishing p2p connections as early as possible to save node resource.
• using some kind of genesis hash or other info to distinguish those node which using different genesis file, prevent node with different genesis file from establishing p2p connections as early as possible to save node resource.

How to Reproduce
Steps to reproduce the behavior:

1. As a genesis validator which use a new genesis file and new codes.

Expected behavior OR Outcome of test
Node can only establish p2p connection with node which has same code and same genesis.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AliYun
• CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
• RAM: 4G
• Disk space: 500G SSD

Additional context
log shows:
V/11 4/1/40 peers ⬇ 0.8kiB/s ⬆ 0.7kiB/s 0.00 bps 0 gas/s CPU: 0%, Mem: 304.1 MiB
[2019-11-20T03:08:33Z ERROR near_client::client_actor] Error processing sync blocks: Old Block
Cause: Unknown
Backtrace:
[2019-11-20T03:08:33Z WARN near_client::client_actor] Banning node for sending invalid block headers
[2019-11-20T03:08:39Z INFO near_client::info] # 55752 Downloading headers 74% V/11 4/1/40 peers ⬇ 0.8kiB/s ⬆ 0.7kiB/s 0.00 bps 0 gas/s CPU: 1%, Mem: 304.1 MiB
[2019-11-20T03:08:48Z ERROR near_client::client_actor] Error processing sync blocks: Old Block
Cause: Unknown
Backtrace:
[2019-11-20T03:08:48Z WARN near_client::client_actor] Banning node for sending invalid block headers

Screenshots Optional
If applicable, add screenshots to help explain your problem.

[TEST][bug] Error running ./start-stakewars.py

Describe the bug or test
Unable to run ./scripts/start_stakewars.py script as per instructions.

To Reproduce
Steps to reproduce the behavior:

1. Run ./scripts/start-stakewars.py --init --account-id=stakehost --signer-keys
2. Copy in account.csv into ~/.near
3. Run ./scripts/start-stakewars.py --acount-id=stakehost

Expected behavior OR Outcome of test
Node should run!

Actual outcome

./start_stakewars.py --account-id=stakehost
****************************************************
* Running NEAR validator node for Stake Wars *
****************************************************
Creating genesis...
Genesis created
Starting NEAR client and Watchtower dockers...
Error response from daemon: No such container: watchtower
Error: No such container: watchtower
Error response from daemon: No such container: nearcore
Error: No such container: nearcore
docker: Error response from daemon: driver failed programming external connectivity on endpoint nearcore (4e60407d2ebf6c0f607b7a9b7c00c188225a703a7055956e8a1dc9c833213218):  (iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 24567 -j ACCEPT: iptables: No chain/target/match by that name.
 (exit status 1)).
Traceback (most recent call last):
  File "./start_stakewars.py", line 35, in <module>
    start_stakewars(args.home, not args.debug, nodocker, args.image, args.verbose)
  File "/data/near/nearcore/scripts/nodelib.py", line 287, in start_stakewars
    run_docker(image, home, '', verbose)
  File "/data/near/nearcore/scripts/nodelib.py", line 139, in run_docker
    envs + [image])
  File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['docker', 'run', '-d', '-p', u'3030:3030', '-p', u'24567:24567', '-v', '/root/.near/:/srv/near', '-v', '/tmp:/tmp', '--ulimit', 'core=-1', '--name', 'nearcore', '--restart', 'unless-stopped', '-e', 'BOOT_NODES=', 'nearprotocol/nearcore:stakewars']' returned non-zero exit status 125

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: Physical/Dedicated
• CPU: 2x Xeon
• RAM: 64GB
• Disk space: 2TB
• Docker 19.03.4

"nearshell creating configuration params folder on current directory" may cause users to be confused

nearshell always creates a folder of configuration named "neardev" on current directory after running "near login" command , when running nearshell on another folder without "neardev" folder, nearshell's login state would change to unlog, which would cause users to be confused .

[BUG] Incompatible Genesis Block

Just pulled the latest nearcore and stakewars:

[2019-12-10T16:26:09Z INFO near] Did not find "/home/j/.near/data" path, will be creating new store database
[2019-12-10T16:26:09Z INFO near_network::peer_manager] Server listening at ed25519:[email protected]:24567
[2019-12-10T16:26:09Z ERROR near_network::peer] Attempting to connect to a node (ed25519:[email protected]:24567@far) with a different genesis block. Our genesis: GenesisId { chain_id: "stakewars", hash: 8ckNyKWhM7BfzB3BhCcy1U1Wd6CAqt7k1Rdabcbiaprx }, their genesis: GenesisId { chain_id: "stakewars", hash: FQDS3eBPH4N4PYPvf6NiWzM41i21hauKmAT45SkpFm84 }
[2019-12-10T16:26:10Z ERROR near_network::peer] Attempting to connect to a node (ed25519:[email protected]:24567@far7) with a different genesis block. Our genesis: GenesisId { chain_id: "stakewars", hash: 8ckNyKWhM7BfzB3BhCcy1U1Wd6CAqt7k1Rdabcbiaprx }, their genesis: GenesisId { chain_id: "stakewars", hash: FQDS3eBPH4N4PYPvf6NiWzM41i21hauKmAT45SkpFm84 }
[2019-12-10T16:26:10Z ERROR near_network::peer] Attempting to connect to a node (ed25519:[email protected]:24567@far4) with a different genesis block. Our genesis: GenesisId { chain_id: "stakewars", hash: 8ckNyKWhM7BfzB3BhCcy1U1Wd6CAqt7k1Rdabcbiaprx }, their genesis: GenesisId { chain_id: "stakewars", hash: FQDS3eBPH4N4PYPvf6NiWzM41i21hauKmAT45SkpFm84 }
[2019-12-10T16:26:10Z ERROR near_network::peer] Attempting to connect to a node (ed25519:[email protected]:24567@far3) with a different genesis block. Our genesis: GenesisId { chain_id: "stakewars", hash: 8ckNyKWhM7BfzB3BhCcy1U1Wd6CAqt7k1Rdabcbiaprx }, their genesis: GenesisId { chain_id: "stakewars", hash: FQDS3eBPH4N4PYPvf6NiWzM41i21hauKmAT45SkpFm84 }
[2019-12-10T16:26:10Z ERROR near_network::peer] Attempting to connect to a node (ed25519:[email protected]:24567@far5) with a different genesis block. Our genesis: GenesisId { chain_id: "stakewars", hash: 8ckNyKWhM7BfzB3BhCcy1U1Wd6CAqt7k1Rdabcbiaprx }, their genesis: GenesisId { chain_id: "stakewars", hash: FQDS3eBPH4N4PYPvf6NiWzM41i21hauKmAT45SkpFm84 }
[2019-12-10T16:26:11Z ERROR near_network::peer] Attempting to connect to a node (ed25519:[email protected]:24567@far2) with a different genesis block. Our genesis: GenesisId { chain_id: "stakewars", hash: 8ckNyKWhM7BfzB3BhCcy1U1Wd6CAqt7k1Rdabcbiaprx }, their genesis: GenesisId { chain_id: "stakewars", hash: FQDS3eBPH4N4PYPvf6NiWzM41i21hauKmAT45SkpFm84 }
[2019-12-10T16:26:19Z INFO near_client::info] # 0 Waiting for peers -/7 0/0/40 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 1%, Mem: 19.9 MiB
[2019-12-10T16:26:29Z INFO near_client::info] # 0 Waiting for peers -/7 0/0/40 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 1%, Mem: 23.3 MiB
[2019-12-10T16:26:39Z INFO near_client::info] # 0 Waiting for peers -/7 0/0/40 peers ⬇ 0 B/s ⬆ 0 B/s 0.00 bps 0 gas/s CPU: 1%, Mem: 23.3 MiB
...

[TEST][choose -> bug | application] Title issue...

Describe the bug or test
A clear and concise description of what the bug is.
Stucking at Downloading headers 100% with a 100% cpu and growing mem, then get killed maybe by oomkiller.
To Reproduce
Steps to reproduce the behavior:

1. Started as a validator at genesis block.
2. publish block to #55 and then change to downloading, with state sync sometimes.
3. 7.5 hours later, cpu goes to 100% and mem goes to 7.2Gb.
4. process was killed and restarted by docker many times during last 7 hours.

Expected behavior OR Outcome of test
more smoothly with a stable cpu and mem usage.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: Ali Cloud
• CPU: 2 cpu
• RAM: 8 G
• Disk space: 200GB

Additional context
[2019-11-04T17:32:25Z ERROR near_client::sync] State sync: state download for shard 1 timed out in 10 seconds
[2019-11-04T17:32:30Z INFO near_client::info] State 1: header V/38 12/1/40 peers ⬇ 756.7kiB/s ⬆ 1.5MiB/s 0.00 bps 0 gas/s CPU: 100%, Mem: 6.6 GiB
[2019-11-04T17:32:36Z ERROR near_client::sync] State sync: state download for shard 1 timed out in 10 seconds
/usr/local/bin/run.sh: line 24: 26 Killed near --home=${NEAR_HOME} run --boot-nodes=${BOOT_NODES}
*** /usr/local/bin/run.sh exited with status 137.

Screenshots Optional

different genesis block.

Describe the bug or test
[2019-11-04T17:19:38Z ERROR near_network::peer] Attempting to connect to a node (ed25519:[email protected]:24567@anj) with a different genesis block. Our genesis: FptUThFY5f7zfTjbomvySKc2HANevmrWoWRtUGG7BqTe, their genesis: AdN294Sovj5YeZZ8cWMsfrnjt9sWHdngYwb14ksSgcAq

[BUG] Latest Staging Fails on Compiling Near Client

Describe the bug or test
Checked out the latest staging branch and then:

j@j-desktop:~/nearcore$ ./scripts/start_stakewars.py --nodocker

---

• Running NEAR validator node for Stake Wars *

---

cargo 1.40.0-nightly (8b0561d68 2019-09-30)
Compiling near-client v0.1.0 (/home/j/nearcore/chain/client)
error: expected one of ., ;, ?, }, or an operator, found =>
--> chain/client/src/client.rs:1122:62
|
1122 | Err(RuntimeError::UnexpectedIntegerOverflow) => {
| ^^ expected one of ., ;, ?, }, or an operator here

error[E0308]: mismatched types
--> chain/client/src/client.rs:1118:21
|
1110 | / if active_validator {
1111 | | debug!(
1112 | | target: "client",
1113 | | "Recording a transaction. I'm {:?}, {}",
... |
1118 | | NetworkClientResponses::ValidTx
| | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected (), found enum near_network::types::NetworkClientResponses
1119 | | } else {
1120 | | self.forward_tx(tx)
1121 | | }
| | -- help: consider using a semicolon here
| |_________________|
| expected this to be ()
|
= note: expected type ()
found type near_network::types::NetworkClientResponses

error[E0308]: mismatched types
--> chain/client/src/client.rs:1120:21
|
1110 | / if active_validator {
1111 | | debug!(
1112 | | target: "client",
1113 | | "Recording a transaction. I'm {:?}, {}",
... |
1120 | | self.forward_tx(tx)
| | ^^^^^^^^^^^^^^^^^^^ expected (), found enum near_network::types::NetworkClientResponses
1121 | | }
| |_________________- expected this to be ()
|
= note: expected type ()
found type near_network::types::NetworkClientResponses
help: try adding a semicolon
|
1120 | self.forward_tx(tx);
| ^
help: consider using a semicolon here
|
1121 | };
| ^

error: aborting due to 3 previous errors

For more information about this error, try rustc --explain E0308.
error: could not compile near-client.

To learn more, run the command again with --verbose.
Compilation failed, aborting
j@j-desktop:~/nearcore$

[BEND][bug] nearcore Memory Leak

Describe the bug or test
nearcore appears to have a memory leak.

To Reproduce
Steps to reproduce the behavior:

1. Run a node
2. Observe gradual rise in memory usage, especially during State Sync
3. Highest observed memory usage is 26GiB

Expected behavior OR Outcome of test
Memory usage doesn't exceed a reasonable limit (e.g. 4, 8, 16GiB)

Specs (please complete the following information):

• OS: Amazon Linux 2
• Which cloud / on-premise: Cloud (AWS EC2)
• CPU: 8
• RAM: 32
• Disk space: 500GB

Additional context
Add any other context about the problem here.

Screenshots Optional

[BEND][choose -> bug | nearcore] WARN trust_dns_proto::xfer::dns_exchange

Getting sometimes such kind of logs

[2019-11-04T20:59:06Z WARN trust_dns_proto::xfer::dns_exchange] failed to associate send_message response to the sender
[2019-11-04T20:59:06Z WARN trust_dns_proto::xfer::dns_exchange] failed to associate send_message response to the sender

To Reproduce
Steps to reproduce the behavior:

1. Start node in docker
2. Wait for sync

Expected behavior OR Outcome of test
A clear and concise description of what you expected to happen.

Specs (please complete the following information):

• OS: linux ubuntu 18.04
• Which cloud / on-premise: Hetzner
• CPU: 4 cores
• RAM: 16 ram
• Disk space: 160 gb

the V sign for validator does not work

by run http post https://rpc.nearprotocol.com/ jsonrpc=2.0 method=query id=hjbtest 'params:=["validators", ""]', I find that now I am a validator.

"result": {
"current_proposals": [],
"current_validators": [
{
"account_id": "far",
"amount": "23672521607228969029",
"public_key": "ed25519:7rNEmDbkn8grQREdTt3PWhR1phNtsqJdgfV26XdR35QL"
},
{
"account_id": "hjbtest",
"amount": "18000000000000000000",
"public_key": "ed25519:EBdoqeeLze6QEnXYz7ECeh5877Yr9UzGCUm89tsYQtUx"
}
],

but from message printed out by the docker log, I can not find'V' sign for a validator.

****

0 gas/s CPU: 90%, Mem: 997.5 MiB
[2019-10-23T15:15:17Z INFO near_client::info] # 81374 Downloading headers 100% -/3 5/4/40 peers ⬇ 10.4kiB/s ⬆ 61.6kiB/s 0.00 bps 0 gas/s CPU: 100%, Mem: 997.5 MiB
[2019-10-23T15:15:27Z INFO near_client::info] # 81374 Downloading headers 100% -/3 5/4/40 peers ⬇ 10.4kiB/s ⬆ 61.6kiB/s 0.00 bps 0 gas/s CPU: 100%, Mem: 997.5 MiB
[2019-10-23T15:15:37Z INFO near_client::info] # 81374 Downloading headers 100% -/3 5/4/40 peers ⬇ 10.4kiB/s ⬆ 61.6kiB/s 0.00 bps 0 gas/s CPU: 100%, Mem: 997.5 MiB
[2019-10-23T15:15:47Z INFO near_client::info] # 81374 Downloading headers 100% -/3 5/4/40 peers ⬇ 10.4kiB/s ⬆ 61.6kiB/s 0.00 bps 0 gas/s CPU: 100%, Mem: 997.5 MiB

[TEST][bug] near_network: Received invalid data

Describe the bug or test
While running a fully synced tatooine node (not a validator) There are occasional error logs with message Received invalid data [...] from None: failed to fill whole buffer. I see this error 4 times in my node's logs between block 617 (node's first block) and 7612 (current block as of writing this issue). The error doesn't look to affect peer-count, bps, cpu or memory usage. So, I'm not sure why it qualifies as an Error.

To Reproduce
Steps to reproduce the behavior:

1. Near node 0.4.5
2. Run with docker sudo scripts/start_stakewars.py
3. Wait for node to sync
4. Observe the logs

Expected behavior OR Outcome of test
Error log is fixed or quieted.

Specs (please complete the following information):

• OS: Amazon Linux 2
• Which cloud / on-premise: cloud (AWS EC2)
• CPU: 8
• RAM: 32GB
• Disk space: 500GB

Additional context

Example Error Log

[2019-11-12T21:12:24Z ERROR near_network::peer] Received invalid data [0, 4, 0, 0, 0, 0, 155, 143, 172, 247, 179, 153, 207, 37, 39, 43, 63, 69, 137, 236, 219, 136, 32, 246, 136, 18, 136, 186, 74, 47, 66, 166, 236, 26, 241, 144, 37, 32, 1, 247, 95, 220, 72, 29, 192, 45, 224, 4, 199, 170, 122, 89, 23, 2, 175, 251, 130, 161, 197, 232, 129, 129, 139, 222, 130, 29, 107, 167, 2, 61, 221, 229, 89, 149, 1, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 13, 240, 150, 127, 58, 53, 110, 247, 254, 164, 202, 219, 203, 35, 193, 83, 89, 140, 163, 49, 178, 83, 122, 110, 113, 180, 176, 211, 115, 129, 6, 96, 61, 122, 109, 203, 104, 210, 173, 167, 254, 74, 201, 144, 128, 47, 121, 164, 82, 59, 132, 132, 255, 48, 128, 94, 45, 171, 228, 205, 61, 151, 141, 14] from None: failed to fill whole buffer

Near Version

$ near -V
NEAR Protocol Node 0.4.5 (build 57dbfab2-modified)

Docker Version

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
 Built:             Mon Jul  1 18:51:44 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a/18.06.1-ce
  Built:            Mon Jul  1 18:53:20 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[BUG] docker error at running genesis-csv-to-json

Describe the bug or test
When exec start_stakewars.py, it shows error:
error: Found argument '--tracked-shards' which wasn't expected, or isn't valid in this context

Here is the detail info
root@buildlinks-test-platform:~/near_stakewars/nearcore# docker logs -f 383714fe3e5d
Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
Running /etc/my_init.d/10_syslog-ng.init...
Nov 20 13:22:04 383714fe3e5d syslog-ng[13]: syslog-ng starting up; version='3.13.2'
Booting runit daemon...
Runit started as PID 21
Running genesis-csv-to-json --home=/srv/genesis-csv-to-json --chain-id=stakewars --tracked-shards=...
error: Found argument '--tracked-shards' which wasn't expected, or isn't valid in this context
USAGE:
genesis-csv-to-json --chain-id --home
For more information try --help
genesis-csv-to-json exited with status 1.
Shutting down runit daemon (PID 21)...

How to Reproduce
Steps to reproduce the behavior:

1. clean .near
2. ./scripts/start_stakewars.py --init --signer-keys --account-id=<your_account_id>
3. downlaod and move accounts.csv into ~/.near folder
4. Run scripts/start_stakewars.py

Expected behavior OR Outcome of test
start node successfully.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AliYun
• CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
• RAM: 4G
• Disk space: 500G SSD

Additional context
docker image info is here:
root@buildlinks-test-platform:~/near_stakewars/nearcore/scripts# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nearprotocol/nearcore stakewars 040fdad633e6 42 hours ago 234MB
nearprotocol/nearcore latest 4b24bba194ad 2 days ago 234MB
v2tec/watchtower latest 3069a9fb302a 20 months ago 9.49MB

Screenshots Optional
If applicable, add screenshots to help explain your problem.

[ BUG | TEST] Tatooine will report username already exists after account creation

Describe the bug or test
On testnet wallet https://wallet.tatooine.nearprotocol.com/create, user cannot create new account for sometimes.

[TEST][bug] Node Stuck

Describe the bug or test

2019-11-13T06:53:09 -- At height 35308, downloading blocks 99%:
memory allocation of 144703488 bytes failed/usr/local/bin/run.sh: line 25: 25 Aborted (core dumped) near --home=${NEAR_HOME} run --telemetry-url=${TELEMETRY_URL} --boot-nodes=${BOOT_NODES}

2019-11-13T06:54:33 -- After that, switch to 35315, downloading blocks 99%:
[2019-11-13T06:55:30Z ERROR near_client::client_actor] State sync received hash 3W6qSJG8Q1n3voj8X8YHfuDRA4Y2hA2Qro6rcFKRCeLD that we're not expecting, potential malicious peer

2019-11-13T06:55:33 -- 35315 Downloading headers 100%, followed by many more errors of:
[2019-11-13T06:55:30Z ERROR near_client::client_actor] State sync received hash 3W6qSJG8Q1n3voj8X8YHfuDRA4Y2hA2Qro6rcFKRCeLD that we're not expecting, potential malicious peer

2019-11-13T06:55:52 -- as well as:
[2019-11-13T06:55:52Z ERROR near_client::sync] State sync: state download for shard 0 timed out in 10 seconds
[2019-11-13T06:55:52Z ERROR near_client::sync] State sync: state download for shard 2 timed out in 10 seconds
[2019-11-13T06:55:52Z ERROR near_client::sync] State sync: state download for shard 4 timed out in 10 seconds
[2019-11-13T06:55:52Z ERROR near_client::sync] State sync: state download for shard 5 timed out in 10 seconds

2019-11-13T07:06:13 -- after which the node finally stuck at
35418 4Mpe2kB6EiojwdFgW2FC29hXek6QK8oFJydwQMiS3VTw V/11 4/3/40

How to Reproduce
Core dump happened at 2019-11-13T06:53:09

Expected behavior OR Outcome of test
Network continuance

Specs:

• OS: Ubuntu 18.04.3 LTS
• Cloud: Custom
• CPU: 2 Cores
• RAM: 12 GB
• Disk space: 100 GB

Additional context

Found additional core dump at 2019-11-13T06:39:34, followed by hundreds of warnings of:

[2019-11-13T06:41:19Z WARN near_network::peer] Received Block while Connecting from Outbound connection.

Sent using near send shows error, but shown as successful in explorer

Story

near send validators_online jleony 1000000000000000000
[WARNING] Didn't find config at /home/<id>/src/config, using default shell config
{ networkId: 'tatooine',
  nodeUrl: 'https://rpc.tatooine.nearprotocol.com',
  contractName: 'near-hello-devnet',
  walletUrl: 'https://wallet.tatooine.nearprotocol.com' }
Sending 1000000000000000000 NEAR to jleony from validators_online
Error:  { Error: send_tx_commit has timed out
    at JsonRpcProvider.sendJsonRpc (/usr/lib/node_modules/near-shell/node_modules/nearlib/lib/providers/json-rpc-provider.js:60:23)
    at process._tickCallback (internal/process/next_tick.js:68:7) type: 'TimeoutError' }

error as above, but shown as successful in explorer   
    https://explorer.tatooine.nearprotocol.com/transactions/B1Xx632yEyAjzm9CXYz1wuzfdqEShyQJ6afRvvUUFFG5

balance in wallet GUI is updated as like the transfer was successful

near send was run in home folder where i have neardev folder that contain private key for validatrs_online id

Missing Crates

Describe the bug or test
Setup a new copy of nearcore via git clone nearcore and staging checkout on a new machine for Stake Wars and the compilation fails due to missing crates.

How to Reproduce
j@andromeda:~/nearcore$ cat ../stakewars.sh

#!/bin/bash
rm -rf ~/.near
mkdir ~/.near
cp ~/accounts.csv ~/.near
cd ~/nearcore
./scripts/start_stakewars.py --signer-keys --init --account-id=joesixpack
sudo chown -R j:j ~/.near
zip -9 ~/stakewarsconfig.zip ~/.near/*
./scripts/start_stakewars.py --nodocker

Expected behavior OR Outcome of test
Should compile succesfully and launch.

Specs (please complete the following information):

• OS: Ubuntu 18.04 LTS
• On-Premise
• CPU: 6
• RAM: 16
• Disk space: 300

Screenshots Optional
j@andromeda:~/nearcore$ ./scripts/start_stakewars.py --nodocker --verbose

---

• Running NEAR validator node for Stake Wars *

---

cargo 1.40.0-nightly (8b0561d68 2019-09-30)
Compiling libc v0.2.65
Compiling cfg-if v0.1.10
Compiling cc v1.0.46
Compiling lazy_static v1.4.0
Compiling semver-parser v0.7.0
Compiling byteorder v1.3.2
error[E0463]: can't find crate for std

error: aborting due to previous error

For more information about this error, try rustc --explain E0463.
error[E0463]: can't find crate for std

error: aborting due to previous error

For more information about this error, try rustc --explain E0463.
error: could not compile libc.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for core

error: could not compile byteorder.
warning: build failed, waiting for other jobs to finish...
error: aborting due to previous error

For more information about this error, try rustc --explain E0463.
error[E0463]: can't find crate for std

error: could not compile lazy_static.
warning: build failed, waiting for other jobs to finish...
error: aborting due to previous error

For more information about this error, try rustc --explain E0463.
error: could not compile semver-parser.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for core

error: aborting due to previous error

For more information about this error, try rustc --explain E0463.
error: could not compile cfg-if.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for std

error: aborting due to previous error

For more information about this error, try rustc --explain E0463.
error: could not compile cc.

To learn more, run the command again with --verbose.
Compilation failed, aborting
j@andromeda:~/nearcore$

[BUG]cargo path missing in nodelib.py

line 50 of nodelib.py:
[os.path.expanduser('cargo'), 'build'] + flags)
should be changed to:
[os.path.expanduser('~/.cargo/bin/cargo'), 'build'] + flags)

when running the following command by sudo, a normal privilege user's PATH does not be passed to root,so a absoult PATH must be specified before 'cargo'

sudo ./scripts/start_stakewars.py --init --signer-keys --account-id=<your_account_id> --nodocker

otherwise, an error of "no such file" will occure.

Block Production Tracking Delay

j@j-desktop:~/nearcore$ RUST_BACKTRACE=full ./scripts/start_testnet.py --nodocker

---

• Running NEAR validator node for Official TestNet *

---

cargo 1.40.0-nightly (8b0561d68 2019-09-30)
Finished release [optimized] target(s) in 0.15s
Using existing node configuration from /home/j/.near/ for testnet
Stake for user 'joesixpack' with '[deleted]'
Starting NEAR client...
Autoupdate is not supported at the moment for runs outside of docker
thread 'main' panicked at 'Failed to deserialize config: Error("missing field block_production_tracking_delay", line: 51, column: 3)', src/libcore/result.rs:1165:5
stack backtrace:
0: 0x56087fcf970c - backtrace::backtrace::libunwind::trace::h19774dea45f5192e
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/libunwind.rs:88
1: 0x56087fcf970c - backtrace::backtrace::trace_unsynchronized::h17947fc1cb72b217
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/mod.rs:66
2: 0x56087fcf970c - std::sys_common::backtrace::_print_fmt::h7b3edf3a4bfcedd3
at src/libstd/sys_common/backtrace.rs:76
3: 0x56087fcf970c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h3ea3134b71160a75
at src/libstd/sys_common/backtrace.rs:60
4: 0x56087f4f239c - core::fmt::write::hb50f65a11f629d84
at src/libcore/fmt/mod.rs:1028
5: 0x56087fcf8f76 - std::io::Write::write_fmt::h30970d8ac223811c
at src/libstd/io/mod.rs:1412
6: 0x56087fcf8bd0 - std::sys_common::backtrace::_print::h48f10c5c50d0e0d7
at src/libstd/sys_common/backtrace.rs:64
7: 0x56087fcf8bd0 - std::sys_common::backtrace::print::h93ea115513ee061c
at src/libstd/sys_common/backtrace.rs:49
8: 0x56087fcf8bd0 - std::panicking::default_hook::{{closure}}::hbf5672c85fe44d10
at src/libstd/panicking.rs:196
9: 0x56087fcf836f - std::panicking::default_hook::h5d2ce66938b54bc6
at src/libstd/panicking.rs:210
10: 0x56087fcf836f - std::panicking::rust_panic_with_hook::h60898b7fda675e17
at src/libstd/panicking.rs:473
11: 0x56087fcf7e9f - std::panicking::continue_panic_fmt::h46da20a693d55410
at src/libstd/panicking.rs:380
12: 0x56087fd07906 - rust_begin_unwind
at src/libstd/panicking.rs:307
13: 0x56087f4ec7b9 - core::panicking::panic_fmt::h4ddf3c5f8367e071
at src/libcore/panicking.rs:84
14: 0x56087f4f2996 - core::result::unwrap_failed::h5541759cb38b9dc1
at src/libcore/result.rs:1165
15: 0x56087f669851 - near::config::Config::from_file::hf82d0cfb0eef19f9
16: 0x56087f36dacd - near::main::h60e53da9b73414fc
17: 0x56087f35e893 - std::rt::lang_start::{{closure}}::hcabdef8a38265d1e
18: 0x56087f377cbe - main
19: 0x7f489495eb6b - __libc_start_main
20: 0x56087f35e7aa - _start
21: 0x0 -
j@j-desktop:~/nearcore$

BUG | Cannot create a new wallet on tatooine

Describe the bug or test
https://wallet.tatooine.nearprotocol.com/create/ not working to create an account

How to Reproduce
Steps to reproduce the behavior:
https://www.dropbox.com/s/wixecbru9zilamk/EyuBkg1xrN.mp4?dl=0

Expected behavior OR Outcome of test
I can create an account

Specs (please complete the following information):

• OS: Windows
• Which cloud / on-premise: Laptop
• CPU: i7
• RAM: 16GB
• Disk space:

[TEST] Require an elegant way to let genesis validator to restake after kicked off for some reason

Describe the bug or test
Say node Alice is a validator in genesis file. When she is kicked off from validator set for some reason, What shall she do to rejoin the validator set? Cause, the normal way of staking is to using near-shell tool with near stake xxx followed by near login. But, as a genesis validator, her signer key (let's call signer accountid Alice_Signer ) is not generated through online wallet. Generally, Alice_Signer is generated through --signer_keys option in python script and there is only a json key file with accountid, public key and secret key. Alice can NOT import her Alice_Signer to online wallet for no Mnemonic phrase for the account in Alice hand.

So, near login are invalid in this scenario. Altough, she can add keyPath and masterAccount params to the near-shell tool, it is not good that NOT able to opeation account Alice_Signer through online wallet.

My suggestion is, either add Mnemonic words in python script tools or let online wallet can import secret key directly.

How to Reproduce
Steps to reproduce the behavior:

1. Be a genesis validator
2. Start your node at a poor network env to let it be kicked from validator set by protocol
3. Then, find a way to rejoin validator set.
4. You will find it is not very convenient.

Expected behavior OR Outcome of test
I hope there would be an elegant way to let genesis validators can oprate their accounts through online wallet.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: AliYun
• CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
• RAM: 4G
• Disk space: 500G SSD

Additional context
near-shell version: 0.14.2

Screenshots Optional
N/A

[BUG] Unconfigured environment 'tatooine'

Describe the bug or test
near-shell doesn't understand tatooine env variable.

How to Reproduce
sudo npm install -g near-shell
npx create-near-app nearstaking
cd nearstaking
export NODE_ENV=tatooine
near login

Expected behavior OR Outcome of test
j@j-desktop:~/nearstaking$ near login
[WARNING] Didn't find config at /home/j/nearstaking/src/config, using default shell config
/usr/local/lib/node_modules/near-shell/blank_project/src/config.js:43
throw Error(Unconfigured environment '${env}'. Can be configured in src/config.js.);
^

Error: Unconfigured environment 'tatooine'. Can be configured in src/config.js.
at getConfig (/usr/local/lib/node_modules/near-shell/blank_project/src/config.js:43:23)
at getConfig (/usr/local/lib/node_modules/near-shell/get-config.js:11:72)
at Object. (/usr/local/lib/node_modules/near-shell/bin/near:178:38)
at Module._compile (internal/modules/cjs/loader.js:1063:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1103:10)
at Module.load (internal/modules/cjs/loader.js:914:32)
at Function.Module._load (internal/modules/cjs/loader.js:822:14)
at Function.Module.runMain (internal/modules/cjs/loader.js:1143:12)
at internal/main/run_main_module.js:16:11
j@j-desktop:~/nearstaking$

Specs (please complete the following information):

• OS: Ubuntu 18.04.3
• CPU: 4 cores
• RAM: 4GB
• Disk space: 110GB

[BEND][bug] State Sync issues

Describe the bug or test
State Sync often times out. Additionally, several

Received Sync while Connecting from Outbound connection.

messages spam stdout shortly after node startup and then intermittently.

To Reproduce
Steps to reproduce the behavior:

1. Run a node

Expected behavior OR Outcome of test
State sync runs more reliably

Specs (please complete the following information):

• OS: Amazon Linux 2
• Which cloud / on-premise: Cloud (AWS EC2)
• CPU: 8
• RAM: 32
• Disk space: 500GB

Additional context

Screenshots Optional

instructions for full_pks

Or even better giving it as output, like we had last Friday.

Different result in runing docker and runing without docker

The error below occured where docker is runing:

================================================================
[2019-10-24T03:00:59Z INFO near_client::info] # 23356 Downloading blocks 99% -/2 3/3/40 peers ⬇ 215.6kiB/s ⬆ 2.3kiB/s 0.00 bps 0 gas/s CPU: 8%, Mem: 164.3 MiB
[2019-10-24T03:01:09Z INFO near_client::info] # 23356 Downloading blocks 99% -/2 3/2/40 peers ⬇ 213.9kiB/s ⬆ 2.3kiB/s 0.00 bps 0 gas/s CPU: 6%, Mem: 164.3 MiB
[2019-10-24T03:01:19Z INFO near_client::info] # 23356 Downloading blocks 99% -/2 3/3/40 peers ⬇ 222.1kiB/s ⬆ 2.4kiB/s 0.00 bps 0 gas/s CPU: 7%, Mem: 164.3 MiB
[2019-10-24T03:01:29Z INFO near_client::info] # 23356 Downloading blocks 99% -/2 3/2/40 peers ⬇ 222.8kiB/s ⬆ 2.4kiB/s 0.00 bps 0 gas/s CPU: 6%, Mem: 164.3 MiB
[2019-10-24T03:01:39Z INFO near_client::info] # 23356 Downloading blocks 99% -/2 3/1/40 peers ⬇ 224.5kiB/s ⬆ 2.5kiB/s 0.00 bps 0 gas/s CPU: 7%, Mem: 164.3 MiB
[2019-10-24T03:01:47Z ERROR near_client::sync] State sync: block request for CKMXpbCHMZuppFzHmVJobZJokX7Q4KuxS6rb3tPicfoM timed out in 10 seconds
[2019-10-24T03:01:49Z INFO near_client::info] State 5: header1: header0: header2: header3: header6: header7: header4: header -/2 3/2/40 peers ⬇ 232.1kiB/s ⬆ 2.5kiB/s 0.00 bps 0 gas/s CPU: 6%, Mem: 164.3 MiB
[2019-10-24T03:01:57Z ERROR near_client::sync] State sync: state download for shard 0 timed out in 10 seconds
[2019-10-24T03:01:57Z ERROR near_client::sync] State sync: state download for shard 1 timed out in 10 seconds
[2019-10-24T03:01:57Z ERROR near_client::sync] State sync: state download for shard 2 timed out in 10 seconds
[2019-10-24T03:01:57Z ERROR near_client::sync] State sync: state download for shard 3 timed out in 10 seconds
[2019-10-24T03:01:57Z ERROR near_client::sync] State sync: state download for shard 4 timed out in 10 seconds
[2019-10-24T03:01:57Z ERROR near_client::sync] State sync: state download for shard 5 timed out in 10 seconds
[2019-10-24T03:01:57Z ERROR near_client::sync] State sync: state download for shard 6 timed out in 10 seconds
[2019-10-24T03:01:57Z ERROR near_client::sync] State sync: state download for shard 7 timed out in 10 seconds
..................................
ignor same message here
..................................
[2019-10-24T03:04:29Z INFO near_client::info] State 5: done1: parts0: done2: done3: parts6: parts7: parts4: done -/2 4/4/40 peers ⬇ 180.6kiB/s ⬆ 0.7kiB/s 0.00 bps 0 gas/s CPU: 8%, Mem: 184.0 MiB
[2019-10-24T03:04:37Z ERROR near_client::sync] State sync: state download for shard 3 timed out in 10 seconds
[2019-10-24T03:04:37Z ERROR near_client::sync] State sync: state download for shard 1 timed out in 10 seconds
[2019-10-24T03:04:39Z INFO near_client::info] State 5: done1: parts0: done2: done3: parts6: done7: done4: done -/2 4/3/40 peers ⬇ 238.0kiB/s ⬆ 0.7kiB/s 0.00 bps 0 gas/s CPU: 9%, Mem: 185.5 MiB
[2019-10-24T03:04:49Z ERROR near_client::sync] State sync: block request for DDFNLax7bgEGmQ7cbZcs3z5KiCYJRwJnjcbkgje7grAB timed out in 10 seconds
[2019-10-24T03:04:49Z INFO near_client::info] State 3: header4: header7: header5: header1: header0: header6: header2: header -/2 4/4/40 peers ⬇ 267.6kiB/s ⬆ 0.4kiB/s 0.00 bps 0 gas/s CPU: 4%, Mem: 185.5 MiB
/usr/local/bin/run.sh: line 24: 30 Illegal instruction (core dumped) near --home=${NEAR_HOME} run --boot-nodes=${BOOT_NODES}
*** /usr/local/bin/run.sh exited with status 132.
*** Shutting down runit daemon (PID 25)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
Oct 24 03:04:57 f7c63ad81b12 syslog-ng[16]: syslog-ng shutting down; version='3.13.2'
*** Killing all processes...

================================================================

**Then I stopped the container and runed nearcore with --nodocker using the same .near folder, It could work well and the "V" sign appeared properly.

I don't know what reason cause these,but I think may it is a bug of docker images.**

================================================================
[2019-10-24T04:06:02Z INFO near_client::info] # 24531 HVqL3ZAmAE7TRzPRgM6Kj5KusiBVtwpn8rd4eyDtmb5a V/1 6/1/40 peers ⬇ 24.1kiB/s ⬆ 83.4kiB/s 1.00 bps 0 gas/s CPU: 6%, Mem: 159.0 MiB
[2019-10-24T04:06:12Z INFO near_client::info] # 24541 9kz4HfH2xZ2h8q8yo1YXTMG6ZqL3pPbZPqhCdMoBhz56 V/1 6/1/40 peers ⬇ 23.8kiB/s ⬆ 82.8kiB/s 1.00 bps 0 gas/s CPU: 7%, Mem: 160.3 MiB
[2019-10-24T04:06:22Z INFO near_client::info] # 24550 68cGhL2qZMmGQRDFbuGc5b2nghZxo23b32V5Vrrn7kqb V/1 6/1/40 peers ⬇ 24.4kiB/s ⬆ 84.1kiB/s 0.90 bps 0 gas/s CPU: 6%, Mem: 160.3 MiB
[2019-10-24T04:06:32Z INFO near_client::info] # 24560 64JpHZ1SqQS846BqPZ4Z8LNoEyfbgNSNDj1ty43M3GUm V/1 6/1/40 peers ⬇ 24.7kiB/s ⬆ 84.9kiB/s 1.00 bps 0 gas/s CPU: 7%, Mem: 160.4 MiB

[TEST][benchmark] Running a node

Hello everyone!

Welcome to Stake Wars! (Plays unlicensed space music) Dun dun dun duuuuun!

This issue is your entry point to the competition. To get started, add a comment here with [node info] and what benchmark you ran against the node. This test is a freebie so everyone who participates can receive points. A template for this comment is provided below. We want to know the specs of whatever you're running against (not necessarily your personal machine) and a cool benchmark. If you can't think of one, we provided some examples you can copy.

For all future bugs or tests, please create a new issue and use the [TEST], [BEND] or [BREAK] labels when you do.

Comment Template for this P3

Machine(s) specs:

• Which cloud / on-premise
• CPU
• RAM
• Disk space

Cool Benchmark:

• Can be speed, performance, scale or something we haven't seen. It just needs a unit of measurement that provides insight.
  • E.g. TPS of node when state takes 50 gb of storage
  • E.g. TPS of 10 nodes on a separate network

[BUG] tatooine explorer crash again

Describe the bug or test
https://explorer.tatooine.nearprotocol.com/ crash again.
Last block is 7hr ago

Expected behavior OR Outcome of test
The newest block is 47358.

Additional context
It fails a lot of times.

Screenshots Optional

[BEND][application] Don't publish validator IP addresses

Describe the bug or test
I really don't like the idea of publishing the validator IP addresses.
This opens up a couple of potential attack vectors, such as e.g. the obvious DDOS.
Ofc it could also be critical to the vitality of the blockchain if a couple of validators are down.

Proposal
Have a look at ways to at least disguise the validator node IP addresses:
E.g. use boot nodes like Ethereum.

[TEST][benchmark] Use Docker memory limits

Describe the bug or test
Use Docker memory limits to control memory leak #17

To Reproduce
Steps to reproduce the behavior:

1. Run a node with Docker memory limits (e.g. docker run ... --memory=4g ...)

Expected behavior OR Outcome of test
Docker memory usage is capped at a reasonable value (e.g. 1g, 2g, or 4g)

Specs (please complete the following information):

• OS: Amazon Linux 2
• Which cloud / on-premise: Cloud (AWS EC2)
• CPU: 8
• RAM: 32
• Disk space: 500GB

Additional context
limit docker resources.

[TEST][bug]Resolved!--- "No such file or directory" error when runing without docker

Describe the bug or test
After runing command: ./scripts/start_stakewars.py --nodocker
OSError:[Errno 2]["No such file or directory" error when runing without docker

when run "start_stakewars.py --nodocker" first time , the cargo build does not work properly.
the "near" file in ~/nearcore/target/release folder has not been generated, which cause "No such file" error.

Solution:

run command line in ~/nearcore directory: cargo build --release -p near
then the "near"file will be generated correctly and then your node can be started normally!

[TEST] near send not working

Describe the bug or test

got below errors, when try to do near send with near-shell

_Error: { Error: send_tx_commit has timed out
at JsonRpcProvider.sendJsonRpc (/usr/lib/node_modules/near-shell/node_modules/nearlib/lib/providers/json-rpc-provider.js:60:23)
at process.tickCallback (internal/process/next_tick.js:68:7) type: 'TimeoutError' }

account id are valid on the tatooine network
jleonyw4 is created with wallet GUI, with enough balance to send
vow4 is in genesis

How to Reproduce
near state jleonyw4
{ networkId: 'tatooine',
nodeUrl: 'https://rpc.tatooine.nearprotocol.com',
contractName: 'near-hello-devnet',
walletUrl: 'https://wallet.tatooine.nearprotocol.com' }
Account jleonyw4
{ amount: '10000000000000000000',
code_hash: '11111111111111111111111111111111',
locked: '0',
storage_paid_at: 742,
storage_usage: 346 }

near state vow4
{ networkId: 'tatooine',
nodeUrl: 'https://rpc.tatooine.nearprotocol.com',
contractName: 'near-hello-devnet',
walletUrl: 'https://wallet.tatooine.nearprotocol.com' }
Missing public key for vow4 in tatooine
Account vow4
{ amount: '10000000000000000000',
code_hash: '11111111111111111111111111111111',
locked: '10032145920893081018',
storage_paid_at: 0,
storage_usage: 182 }

near send jleonw4 vow4 1000000000000000000
{ networkId: 'tatooine',
nodeUrl: 'https://rpc.tatooine.nearprotocol.com',
contractName: 'near-hello-devnet',
walletUrl: 'https://wallet.tatooine.nearprotocol.com' }
Sending 1000000000000000000 NEAR to vow4 from jleonw4
Error: { Error: send_tx_commit has timed out
at JsonRpcProvider.sendJsonRpc (/usr/lib/node_modules/near-shell/node_modules/nearlib/lib/providers/json-rpc-provider.js:60:23)
at process._tickCallback (internal/process/next_tick.js:68:7) type: 'TimeoutError' }

Expected behavior OR Outcome of test
1 NEAR should be sent from jleonyw4 to vow4

Specs (please complete the following information):

• OS:
• Which cloud / on-premise:
• CPU:
• RAM:
• Disk space:

Additional context
Add any other context about the problem here.

Screenshots Optional
If applicable, add screenshots to help explain your problem.

[test][bug] thread is killed all the time with cpu 100%

Describe the bug or test

[2019-11-05T16:11:28Z INFO  near_client::info] #       0 Downloading headers 100% V/38  4/2/40 peers ⬇ 601.0kiB/s ⬆ 580.3kiB/s 0.00 bps 0 gas/s CPU: 100%, Mem: 3.3 GiB
Nov  5 16:17:03 3910524407ea cron[24]: (CRON) error (can't fork)
Nov  5 17:17:03 3910524407ea cron[24]: (CRON) error (can't fork)
/usr/local/bin/run.sh: line 24:    23 Killed                  near --home=${NEAR_HOME} run --boot-nodes=${BOOT_NODES}
*** /usr/local/bin/run.sh exited with status 137.
*** Shutting down runit daemon (PID 19)...
*** Running /etc/my_init.post_shutdown.d/10_syslog-ng.shutdown...
Nov  5 17:21:02 3910524407ea syslog-ng[13]: syslog-ng shutting down; version='3.13.2'
*** Killing all processes...

To Reproduce
Steps to reproduce the behavior:
when running a node

container is live, but nearcore thread is dead.

root@10-8-63-215:/home/ubuntu# docker ps 
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                                              NAMES
c6cee8334156        v2tec/watchtower                  "/watchtower nearp..."   10 hours ago        Up 10 hours                                                            watchtower
3910524407ea        nearprotocol/nearcore:stakewars   "/sbin/my_init -- ..."   10 hours ago        Up 10 hours         0.0.0.0:3030->3030/tcp, 0.0.0.0:24567->24567/tcp   nearcore

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise:
• CPU: 2C
• RAM: 4G
• Disk space: 50G

[TEST][benchmark] Stucking at downloading header with high cpu and mem usage.

Describe the bug or test

Stucking at Downloading headers 100% with a 100% cpu and growing mem, then get killed maybe by oomkiller.

To Reproduce
Steps to reproduce the behavior:

• Started as a validator at genesis block.
• publish block to #55 and then change to downloading, with state sync sometimes.
• 7.5 hours later, cpu goes to 100% and mem goes to 7.2Gb.
• process was killed and restarted by docker many times during last 7 hours.

Expected behavior OR Outcome of test

• more smoothly with a stable cpu and mem usage.

Specs (please complete the following information):

• OS: Ubuntu 18.04
• Which cloud / on-premise: Ali Cloud
• CPU: 2 cpu
• RAM: 8 G
• Disk space: 200GB

Additional context
[2019-11-04T17:32:25Z ERROR near_client::sync] State sync: state download for shard 1 timed out in 10 seconds
[2019-11-04T17:32:30Z INFO near_client::info] State 1: header V/38 12/1/40 peers ⬇ 756.7kiB/s ⬆ 1.5MiB/s 0.00 bps 0 gas/s CPU: 100%, Mem: 6.6 GiB
[2019-11-04T17:32:36Z ERROR near_client::sync] State sync: state download for shard 1 timed out in 10 seconds
/usr/local/bin/run.sh: line 24: 26 Killed near --home=${NEAR_HOME} run --boot-nodes=${BOOT_NODES}
*** /usr/local/bin/run.sh exited with status 137.

Screenshots Optional

[TEST] accounts.csv in master branch committed 4 hrs ago having genesis time of Nov 26

Describe the bug or test
A clear and concise description of what the bug is.
accounts.csv in master branch committed 4 hrs ago having genesis time of Nov 26

How to Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See thing

Expected behavior OR Outcome of test
A clear and concise description of what you expected to happen.

Specs (please complete the following information):

• OS:
• Which cloud / on-premise:
• CPU:
• RAM:
• Disk space:

Additional context
Add any other context about the problem here.

Screenshots Optional
If applicable, add screenshots to help explain your problem.

[TEST] Log message for when a validator is kicked out

Describe the bug or test
When a validator is kicked out there is no indication of why. There is only the indication that the node is no longer a validator: no more V icon.

How to Reproduce
Steps to reproduce the behavior:

1. Be a validator
2. Get kicked out

Expected behavior OR Outcome of test
A helpful warning or error log when a validator is kicked out with the explanation.

Specs (please complete the following information):

• OS: Amazon Linux 2
• Which cloud / on-premise: Cloud (AWS EC2)
• CPU: 8
• RAM: 32gb
• Disk space: 500gb

Additional context
Account stakedinc5 was a validator in genesis for week 5

Screenshots

related to #63 and #62