Performance issues about multichain-explorer

[gidgreen]

Do you have some Python tool that lets you trace which piece of code it's stuck on?

[roderik]

I can install and configure anything we need to get to the bottom of this, but I have no clue on the python ecosystem.

An educated guess, all these transactions are in one stream. Abe can handle the bitcoin blockchain with 90gb even on sqlite. So probably in the additions for streams that are triggered for the homepage.

[gidgreen]

You should be able to use the basic pdb Python debugger – see the documentation here:

https://docs.python.org/2/library/pdb.html

We haven't tried it yet, but you should be able to add an import pdb; pdb.set_trace() at the top of the main Mce/abe.py file, then re-run the installation instructions in the Explorer README, then use the continue command in the debugger to let the Explorer start running, then when it looks stuck, use the where or list commands to see which piece of code it's stuck on.

[bitcartel]

@roderik Are you using dummy/test data? If yes, maybe you could share all the chain and explorer data so we can try and replicate.

[easeev]

Have a similar issue.

Debugged a little:

2019-09-20 10:11:31.953006
getting num_txs
2019-09-20 10:11:31.985604
getting num_addresses
2019-09-20 10:16:05.755095 <- took almost 5 mins
getting num_peers
2019-09-20 10:16:05.821592
getting num_assets
2019-09-20 10:16:05.876624
getting num_streams
2019-09-20 10:16:21.742359
got all nums
2019-09-20 10:16:21.743385
getting mempool
2019-09-20 10:16:21.791694
getting recenttx
2019-09-20 10:17:47.790923 <- took more than 1 min
getting sorted_mempool
127.0.0.1 - - [20/Sep/2019 10:17:47] "GET / HTTP/1.1" 200 6175

These queries look suboptimal when you have a significant amount of transactions (as any DISTINCT query on a complex table/view without proper indexes would be):

multichain-explorer/Mce/DataStore.py

Line 3734 in 4446685

SELECT COUNT(DISTINCT(pubkey_hash)) FROM txout_detail WHERE chain_id = ?

multichain-explorer/Mce/DataStore.py

Lines 3894 to 3898 in 4446685

	SELECT DISTINCT tx_hash
	FROM txout_detail
	WHERE chain_id=? AND pubkey_id != ?
	ORDER BY block_height DESC, tx_id DESC
	LIMIT ?

In my case it's actually not that large but already quite problematic:

sqlite> select count(*) from txout_detail;
1758971

[easeev]

Did anyone have time to look at this issue? It looks critical for networks with significant number of transactions. @gidgreen @bitcartel

[gidgreen]

This should be fixable by adding the appropriate indexes at the time of table creation. Do you want to try to add these indexes during initialization in DataStore.py, restarting the Explorer afresh, and confirming?

[easeev]

Adding indexes didn't help. Implemented a workaround by using more optimal queries: chainstack#4.