Cron jobs using CodeChain network
codechain-io / cron-jobs Goto Github PK
View Code? Open in Web Editor NEWCron jobs using CodeChain network
License: ISC License
Cron jobs using CodeChain network
License: ISC License
[warn][wc][fee-monitor] finds a problem
{
"server": "corgi",
"moment": {
"tag": "term",
"value": 469870
},
"errors": [
{
"type": "validator",
"account": "wccq9hv8e73frxr2c8txp5snu7gd4pth3z92ganlnz8",
"expected": "9999990104170",
"actual": "9999990104155"
},
{
"type": "validator",
"account": "wccq9gw2hdwfh6et35calxp90lft4qfwmdxpsd2dq5y",
"expected": "9999990124969",
"actual": "9999990124939"
},
{
"type": "validator",
"account": "wccqxmehv99je74c9gfmk9fy4m7qp6rwfcgdq02x7hm",
"expected": "9999990054265",
"actual": "9999990054204"
}
]
}
The cause of the problem is an error when getting previous term's validators.
actual:
sortedValidators [ [ 'wccqxmehv99je74c9gfmk9fy4m7qp6rwfcgdq02x7hm' ],
[ 'wccq9gw2hdwfh6et35calxp90lft4qfwmdxpsd2dq5y' ],
[ 'wccq9hv8e73frxr2c8txp5snu7gd4pth3z92ganlnz8' ] ]
expected:
sortedValidators [ [ 'wccqxmehv99je74c9gfmk9fy4m7qp6rwfcgdq02x7hm',
'wccq9l5ly2ppsddta4gynmy05acxzxp9h2mqvqyzufh'],
[ 'wccq9gw2hdwfh6et35calxp90lft4qfwmdxpsd2dq5y' ],
[ 'wccq9hv8e73frxr2c8txp5snu7gd4pth3z92ganlnz8' ] ]
Currently, fee-monitor is using javascript number type when calculating rewards for validators. But there is a potential hazard of overflow when multiplying a number and dividing later to integrate round-off errors into one.
To check the tool's notification setting is working, a tool needs to send a notification when it starts.
I found this error in the fee monitor in Corgi network.
Here is the log.
BlockNumber: 34926
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 1 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 2 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 3 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 4 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 5 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 6 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 7 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 8 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 9 sec(s)
TxTypes: custom
Too many retries: 10
{ error: Error: U64 must be a positive integer but found 0x
at new U64 (/home/ubuntu/fee-monitor/fee-monitor/node_modules/codechain-primitives/lib/value/U64.js:16:19)
at Function.ensure (/home/ubuntu/fee-monitor/fee-monitor/node_modules/codechain-primitives/lib/value/U64.js:104:47)
at Object.decodeU64 (/home/ubuntu/fee-monitor/fee-monitor/build/Stake.js:16:26)
at DynamicChecker.handleTransactions (/home/ubuntu/fee-monitor/fee-monitor/build/DynamicChecker.js:199:67)
at DynamicChecker.checkBlockDynamic (/home/ubuntu/fee-monitor/fee-monitor/build/DynamicChecker.js:70:20)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7) }
(node:22669) UnhandledPromiseRejectionWarning: Error: U64 must be a positive integer but found 0x
at new U64 (/home/ubuntu/fee-monitor/fee-monitor/node_modules/codechain-primitives/lib/value/U64.js:16:19)
at Function.ensure (/home/ubuntu/fee-monitor/fee-monitor/node_modules/codechain-primitives/lib/value/U64.js:104:47)
at Object.decodeU64 (/home/ubuntu/fee-monitor/fee-monitor/build/Stake.js:16:26)
at DynamicChecker.handleTransactions (/home/ubuntu/fee-monitor/fee-monitor/build/DynamicChecker.js:199:67)
at DynamicChecker.checkBlockDynamic (/home/ubuntu/fee-monitor/fee-monitor/build/DynamicChecker.js:70:20)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
(node:22669) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 21)
(node:22669) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
fee-monitor has been stalled for 30 seconds:{
"blockNumber": 34926,
dynamicChecker
class uses internal states. If a connection error occurs in the middle of fee distribution checking routine, retry logic re-call the function checkBlock
with the parameter dynamicChecker
which has partially updated state. That may leads to an block error and a term error.
Line 161 in 6ed9a56
Script to check whether the block reward is distributed as specified.
Here is the log from the auto-self-nominate.
sdk wc http://127.0.0.1:8080
Best block number: 2523960
Current Term Id: 0
Deposit Supplement: 10,000,000
Self Nomination tx sent: f9b72fbd75d98ea9e423456bd86b24b0861e96b5c3f1d17d12a115d83176179e
Candidate info
TypeError: Cannot read property 'deposit' of undefined
at send (/home/ubuntu/cron-jobs/auto-self-nominate/build/index.js:150:46)
at processTicksAndRejections (internal/process/task_queues.js:85:5)
To calculate nomination deposit returns, fee-monitor had to watch the self-nomination transaction and save the deposit quantity to fee-monitor's own state. But to calculate it correctly, minimum duration of running fee-monitor is about "11 days". Because the path from the state "Jailed" to "Eligible" takes 240 terms to transit.
If update fee-monitor and restart it in the middle, there may be some warnings about balance expectation fails.
I'm suspecting the line below:
cron-jobs/fee-monitor/src/index.ts
Line 200 in eda1e3e
Make a script that sends a report when the balance of accounts are out of range.
And I think it's good if it sends a report daily.
Following the codechain issue#1710, the fee-monitor's logic should be changed correspondingly.
When one tries to run auto-self-nominate script with banned account, currently it send the message : "Your account ${account} is permanently banned".
But It would be better to show the reason why the account is banned.
Currently, there are two ways to set up the configuration, because each tool is written by the various developers.
One is using config
and the other is using environments.
Some developers, including me, prefer config
to environments because the environments vanish when the session closes.
And others prefer environments because it corrupts the git history and it's still hard to track because it depends on environments to choose the configuration file.
But now, we can keep the environment because we use tmuxinator, so I think it's better to unify styles by using environment variables.
Currently, too much notifications are sending to both slack and e-mail. I think we should divide it into slack only notifications and e-mail only notifications.
Each tool's README.md file should have a section about how to update.
if the block sync stopped more than 5 minutes, indexer-monitor sends an error message.
The original intention was 1 hour.
When fee-monitor finds mismatches between fee-monitor's fee distribution and CodeChain's fee distribution, it sends warning messages through slack. But information in those messages are not enough to analyze the problem directly.
Current some error messages only send "Internal error" but it should be more specific.
We need a script to send CCC from a hot wallet to a cold wallet.
We need more information in logs and notifications
(node:7171) UnhandledPromiseRejectionWarning: Error: The number of validators is 2 at 463078. min: 4
at Object.check [as default] (/home/beagle/val-val/val-val/build/election.js:57:15)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:7171) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:7171) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Error: The number of validators is 2 at 463078. min: 4
Error: The number of validators is 2 at 463078. min: 4
at Object.check [as default] (/home/beagle/val-val/val-val/build/election.js:57:15)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:7171) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 1)
The fee monitor said that there is a fee calculation error in the CodeChain Main network.
I checked the stakeholder's balance at the block, there was no error at the block.
There is a transaction with 100 CCC fee and each stakeholder received 10 CCCs.
I copied the error message:
{
"server": "mainnet",
"moment": {
"tag": "block",
"value": 5343247
},
"errors": [
{
"type": "stakeholder",
"account": "cccqyhk4388mymckk40ctf0skgkl8nf89f6sgpzwzke",
"expected": "10010145050",
"actual": "10010145030"
},
{
"type": "stakeholder",
"account": "cccq9g0ayvrvmzcs2qf8yygat0dg9073g5xccd3sydw",
"expected": "10010144950",
"actual": "10010144930"
},
{
"type": "stakeholder",
"account": "cccq9hpz4m5sx0ea42l2hfm8nja0xzzsnx0vv0p7mzw",
"expected": "10010144850",
"actual": "10010144830"
},
{
"type": "stakeholder",
"account": "cccqxgwashdy0fp5cwlvp4ntkzmy3xvjedfjgfzrxf7",
"expected": "10010144850",
"actual": "10010144830"
},
{
"type": "stakeholder",
"account": "cccqxgupxd5j0squ7t8w3etxjf0fn2z4axz0spycqum",
"expected": "10010145250",
"actual": "10010145230"
},
{
"type": "stakeholder",
"account": "cccqx7ycl2awx6awqyjf9ns4532xj7eg63w6v6hlnu0",
"expected": "10010144850",
"actual": "10010144830"
},
{
"type": "stakeholder",
"account": "cccq8r9as2drgp4xwfzz7t0xc9k977y8mw7hus3zq8e",
"expected": "10010144950",
"actual": "10010144930"
},
{
"type": "stakeholder",
"account": "cccq826j0p6fvpk02p0n5th7ukqu5clrnyldcxgrgtz",
"expected": "10010144850",
"actual": "10010144830"
},
{
"type": "stakeholder",
"account": "cccq8dhpcfqt94mx8dntxxd6420782vtd4tpvpgcgjh",
"expected": "10010145250",
"actual": "10010145230"
},
{
"type": "stakeholder",
"account": "cccq8u8se8k00vcqk4pl707qe9v60hs3jj27g4rqkm9",
"expected": "9417937350",
"actual": "9417937330"
}
]
}
I stopped the helicopter because it generates lots of pending transactions.
Fix and rerun it.
Currently, it sends a notification forever when a node dies.
TimeLock test creates a transaction that goes a little far in the future and creates a transaction that can pass right away. However the two transactions are signed by the same account and using the same sequence, the second transaction may not be processed as intended.
I found that fee-monitor has been stalled without throwing error or logging output.
We should add a watchdog to fee-monitor.
Hi @HoOngEe,
cron-jobs/monitor/SlackNotify.ts
Lines 28 to 30 in eb1cb0f
monitor
and fee-monitor
script currently require following environment variables to be set:
SENDGRID_API_KEY
SENDGRID_TO
SLACK_WEBHOOK_URL
There are some cases that the user wants to run the script, but doesn't want to receive notifications. (eg. in local test network)
I think printing warning messages is enough for this case.
Like [email protected]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.