[Bug]: Error get hypeblock 12482351+ about mx-chain-go

[bogdan-rosianu]

Hello @oggy95 !
Can you be more specific? Which environment? Mainnet/Testnet/Devnet
Also, by using gateway.elrond.com / api.elrond.com?
Thanks

[oggy95]

In issue I wrote - Mainnet.
That is deployed own node for connecting to Elrond network.
Node version: v1.3.50
Go version go1.17.6
When using gateway.elrond.com, response is successful for those blocks.

[bogdan-rosianu]

There might be issues with your nodes. Did you set DBLookupExtension to true? Also, make sure that the nodes are in sync with the network. If these 2 are true, then please provide some logs

[oggy95]

I setup node via elrond-go-scripts. How can I check if DBLookupExtension is enabled?
And yes, node is synced.

[bogdan-rosianu]

This config flag has to be set to true:
https://github.com/ElrondNetwork/elrond-go/blob/master/cmd/node/config/config.toml#L836

[oggy95]

Set this field to true and restart the node. Didn't work.
journalctl -f -u elrond-proxy.service logs

-- Logs begin at Mon 2022-12-19 05:28:06 UTC. --
Dec 19 09:38:01 EGLD-node proxy[3705]: [GIN] 2022/12/19 - 09:38:01 | 200 |     664.972µs |  157.90.250.214 | GET      "/network/status/4294967295"
Dec 19 09:38:26 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:26.438]   block request                            shard id = 4294967295 nonce = 12482351 observer = http://127.0.0.1:8083
Dec 19 09:38:26 EGLD-node proxy[3705]: ERROR[2022-12-19 09:38:26.532]   block request                            observer = http://127.0.0.1:8081 error = {"data":null,"error":"getting block failed: key not found","code":"internal_issue"}
Dec 19 09:38:26 EGLD-node proxy[3705]: WARN [2022-12-19 09:38:26.532]   [internal error] api request             path = /hyperblock/by-nonce/12482351 duration = 816.542669ms status = 500 client IP = 95.217.152.178 request =  response = {"data":null,"error":"sendingrequesterror","code":"internal_issue"}
Dec 19 09:38:26 EGLD-node proxy[3705]: [GIN] 2022/12/19 - 09:38:26 | 500 |  817.137631ms |  95.217.152.178 | GET      "/hyperblock/by-nonce/12482351"
Dec 19 09:38:29 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:29.705]   block request                            shard id = 4294967295 nonce = 12482350 observer = http://127.0.0.1:8083
Dec 19 09:38:30 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:30.468]   block request                            shard id = 0 hash = 46ed63d5421c9a0ce795d805928826d6d03af3d98026ce51256734b10656d965 observer = http://127.0.0.1:8080
Dec 19 09:38:31 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:31.284]   block request                            shard id = 2 hash = 87e460c4b5d83c58230b299e9db4517eba263c95053030a05dc0528314cf25e1 observer = http://127.0.0.1:8082
Dec 19 09:38:31 EGLD-node proxy[3705]: WARN [2022-12-19 09:38:31.285]   [too long] api request                   path = /hyperblock/by-nonce/12482350 duration = 1.592095784s status = 200 client IP = 95.217.152.178 request =  response = {"data":{"hyperblock":{"hash":"7d33068bd0515c002a0c2e61d1fa094ac0274a1d02e8b75455dce077f58bbc81","prevBlockHash":"8c9fdf08de75b5f1f3c6a8b9d50f49fcfc0d97a16ab19352393f1a0b902daa11","stateRootHash":"bd7d3afb5456236c621fa482196b707910a2d3440003410d08348d4acf4785e0","nonce":12482350,"round":12502100,"epoch":868,"numTxs":9,"accumulatedFees":"0","developerFees":"0","accumulatedFeesInEpoch":"962893207292...
Dec 19 09:38:31 EGLD-node proxy[3705]: [GIN] 2022/12/19 - 09:38:31 | 200 |  1.592233061s |  95.217.152.178 | GET      "/hyperblock/by-nonce/12482350"
Dec 19 09:38:49 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:49.086]   block request                            shard id = 4294967295 nonce = 12482351 observer = http://127.0.0.1:8083
Dec 19 09:38:49 EGLD-node proxy[3705]: ERROR[2022-12-19 09:38:49.103]   block request                            observer = http://127.0.0.1:8081 error = {"data":null,"error":"getting block failed: key not found","code":"internal_issue"}
Dec 19 09:38:49 EGLD-node proxy[3705]: WARN [2022-12-19 09:38:49.103]   [internal error] api request             path = /hyperblock/by-nonce/12482351 duration = 19.041584ms status = 500 client IP = 95.217.152.178 request =  response = {"data":null,"error":"sendingrequesterror","code":"internal_issue"}
Dec 19 09:38:49 EGLD-node proxy[3705]: [GIN] 2022/12/19 - 09:38:49 | 500 |   19.087861ms |  95.217.152.178 | GET      "/hyperblock/by-nonce/12482351"
Dec 19 09:38:51 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:51.272]   rate limiter map has been reset          version =  time = 2022-12-19 09:38:51.27236208 +0000 UTC m=+309861.106801199
Dec 19 09:38:51 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:51.272]   rate limiter map has been reset          version = v1.0 time = 2022-12-19 09:38:51.272311305 +0000 UTC m=+309861.106750375
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.711]   node status                              address = http://127.0.0.1:8080 shard = 0 nonce = 12550212 probable highest nonce = 12550212 is synced = true is ready for VM Queries = true is fallback = false
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.713]   node status                              address = http://127.0.0.1:8082 shard = 2 nonce = 12550690 probable highest nonce = 12550690 is synced = true is ready for VM Queries = true is fallback = false
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.714]   node status                              address = http://127.0.0.1:8083 shard = 4294967295 nonce = 12534437 probable highest nonce = 12534437 is synced = true is ready for VM Queries = true is fallback = false
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.715]   node status                              address = http://127.0.0.1:8081 shard = 1 nonce = 12493362 probable highest nonce = 12493382 is synced = false is ready for VM Queries = true is fallback = false
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.715]   backup observer updated                  address = http://127.0.0.1:8081 is fallback = false shard = 1
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.715]   shard 0 active nodes                     observers count = 1 addresses = http://127.0.0.1:8080 fallback addresses =
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.715]   shard 1 active nodes                     observers count = 1 addresses = http://127.0.0.1:8081 fallback addresses =
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.715]   shard 2 active nodes                     observers count = 1 addresses = http://127.0.0.1:8082 fallback addresses =
Dec 19 09:38:52 EGLD-node proxy[3705]: INFO [2022-12-19 09:38:52.715]   shard 4294967295 active nodes            observers count = 1 addresses = http://127.0.0.1:8083 fallback addresses =
Dec 19 09:39:00 EGLD-node proxy[3705]: INFO [2022-12-19 09:39:00.277]   validator statistics fetched from API    observer = http://127.0.0.1:8083
Dec 19 09:39:24 EGLD-node proxy[3705]: INFO [2022-12-19 09:39:24.109]   network metrics request                  shard ID = 4294967295 observer = http://127.0.0.1:8083
Dec 19 09:39:24 EGLD-node proxy[3705]: [GIN] 2022/12/19 - 09:39:24 | 200 |    1.914425ms |  157.90.250.214 | GET      "/network/status/4294967295"

./script.sh get_logs
https://dropmefiles.com/jKl4n

[bogdan-rosianu]

Based on the logs, the shard 1 node seems to be stuck for reasons I cannot tell based on the logs you provided. All other nodes seem to run fine

[bogdan-rosianu]

Any update? Did you manage to check the shard 1 node?

[oggy95]

I restarted service for shard 1 node, but it didn't help.
What can I try to do with it or which logs I should provide to give more info for the problem?

[bogdan-rosianu]

Hi! The quick solution would be to perform a fresh start (removing the DB, logs) and starting the squad again. But this would mean losing the Elasticsearch history, which I assume won’t work for you.
What you can try is to remove the latest epoch db for the observer with issues, and restart it as full archive (check prefs.toml file). This way, it would sync from the network the missing parts. Let me know if this works

[iulianpascalau]

Hello, any update on this? Is this solved?

[oggy95]

Sorry, doesn't have a time to look for it.
For now I will close the issue, if solution doesn't work, I will open a new one.

[rvignolo]

hi @iulianpascalau and @bogdan-rosianu, nice meeting you!

Let's assume that I have all the Epochs for all shards for the devnet.

I want to be able to start my observing squad from a specific epoch so it can populate my elastic search instance from that epoch onwards (I don't want it to start from the epoch 0, even if I have the Epoch_0 folder for all shards.

So far, I am able to start my observing squad and it gets the last 4 epochs (by default), synchronises with the network and also populates the elastic search instance. So everything is working.

Now, I want to get not only the last 4 epochs but I want to be able to specify from which Epoch I want to start. Is that even possible? From @bogdan-rosianu comment seems that it is possible:

What you can try is to remove the latest epoch db for the observer with issues, and restart it as full archive (check prefs.toml file). This way, it would sync from the network the missing parts. Let me know if this works

So this implies that I should go inside each ~/elrond-nodes/node-*/db/D/ and put inside the first Epoch_xxxx that I want and it will start from there?

Another question is: should I go instead to this folder ~/go/src/github.com/multiversx/mx-chain-go/cmd/node?

Sorry for all these questions but the documentation is really not clear at all: the import db method (https://docs.multiversx.com/validators/import-db/) is not clear also...

Any help to accomplish this would be much appreciated.

[rvignolo]

update: everything seems to be working after:

• creating the import-db folder inside ~/elrond-nodes/node-0 and putting inside the database downloaded from a provided URL at the validators telegram group.
• run ./node -use-log-view -log-level *:INFO -import-db ./import-db -import-db-no-sig-check inside such folder.
• this is creating the ~/elrond-nodes/node-0/db/D/Epoch_xxxx folders.

However, there are two issues:

• it is starting from Epoch 0, but I don't need to have information that far ago at my elastic search instance
• after almost 50 epochs, it got stuck with this error message:

WARN [2023-02-11 20:38:50.262]   persister create error                   error = open /home/hatom/elrond-nodes/node-0/db/D/Epoch_48/Shard_0/DbLookupExtensions/MiniblocksMetadata/LOCK: too many open files for path /home/hatom/elrond-nodes/node-0/db/D/Epoch_48/Shard_0/DbLookupExtensions/MiniblocksMetadata
WARN [2023-02-11 20:38:50.262]   persister create error                   error = open /home/hatom/elrond-nodes/node-0/db/D/Epoch_47/Shard_0/DbLookupExtensions/MiniblocksMetadata/LOCK: too many open files for path /home/hatom/elrond-nodes/node-0/db/D/Epoch_47/Shard_0/DbLookupExtensions/MiniblocksMetadata
WARN [2023-02-11 20:38:50.262]   persister create error                   error = open /home/hatom/elrond-nodes/node-0/db/D/Epoch_48/Shard_0/DbLookupExtensions/MiniblocksMetadata/LOCK: too many open files for path /home/hatom/elrond-nodes/node-0/db/D/Epoch_48/Shard_0/DbLookupExtensions/MiniblocksMetadata
WARN [2023-02-11 20:38:50.262]   persister create error                   error = open /home/hatom/elrond-nodes/node-0/db/D/Epoch_47/Shard_0/DbLookupExtensions/MiniblocksMetadata/LOCK: too many open files for path /home/hatom/elrond-nodes/node-0/db/D/Epoch_47/Shard_0/DbLookupExtensions/MiniblocksMetadata
WARN [2023-02-11 20:38:50.262]   persister create error                   error = open /home/hatom/elrond-nodes/node-0/db/D/Epoch_48/Shard_0/DbLookupExtensions/MiniblocksMetadata/LOCK: too many open files for path /home/hatom/elrond-nodes/node-0/db/D/Epoch_48/Shard_0/DbLookupExtensions/MiniblocksMetadata
WARN [2023-02-11 20:38:51.044]   leveldb serial putBatch                  error = open /home/hatom/elrond-nodes/node-0/db/D/Static/Shard_0/StatusMetricsStorageDB/000014.log: too many open files
2023-02-11T20:38:53.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:38:58.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:03.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:08.381Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:13.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:18.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:23.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:28.381Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:33.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:38.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}
2023-02-11T20:39:43.380Z	ERROR	basichost	basic/basic_host.go:327	failed to resolve local interface addresses	{"error": "route ip+net: netlinkrib: too many open files"}

Thanks!

[iulianpascalau]

It seems it went back to the Epoch_47 and hit the "too many open files" error. I think you forgot to copy the Static directory, as far as I've understood. That directory is mandatory to be present as it contains data used by the node in each epoch (that is why it is not stored in a Epoch_xxx directory)