insight randomly stops syncing. after deleting the DB and resyncing, it doesn't complete the sync about insight-api

[robby-d]

Deleting the bitcoind testnet data and scping it over from another host, and then rebuilding the insight test DB, seemed to fix this. However, this is concerning that somehow the data just magically got corrupted (this is on a server at a data center, ECC memory, RAID, etc).

I'll monitor things and see if I see the issue with insight just randomly stopping to update the blockchain happening again. Worse case I can try using rpc-based syncing, and/or reverting to an older version of insight (unless you have other recommendations)?

[matiu]

This was a running insight server, or did you stop it  for sometime and the restart it?—
Sent from Mailbox for iPhone

On Tue, Mar 18, 2014 at 8:24 PM, xnova [email protected] wrote:

Deleting the bitcoind testnet data and scping it over from another host, and then rebuilding the insight test DB, seemed to fix this. However, this is concerning that somehow the data just magically got corrupted (this is on a server at a data center, ECC memory, RAID, etc).

I'll monitor things and see if I see the issue with insight just randomly stopping to update the blockchain happening again. Worse case I can try using rpc-based syncing, and/or reverting to an older version of insight (unless you have other recommendations)?

Reply to this email directly or view it on GitHub:
#17 (comment)

[robby-d]

It was running and had stop updating the blockchain (i.e. blockChainHeight showed an old block number, like 20+ blocks behind, and status was finished, and syncPercentage was 100)... so I stopped it and restarted, and it still didn't update. At that point, I pulled an update from github (to the newest master), reran "npm install" to install the updated deps (e.g. soop), and deleted the insight-api/db/testnet dir (and remade it so that it would just have an "empty" file, as in the repo).

I then restarted insight.... it took a time or two for insight to notice this and start grabbing new blocks, but once it did, it gave this error. ...restarting bitcoind didn't fix it, etc.

The "fix" it seems was to stop bitcoind, delete it's data, pull the testnet blockchain data from another system, and restart bitcoind with that data, then totally delete the insight data and restart it, and it downloaded the testnet data from bitcoind (using the files method, not RPC) and seemed to sync up to 100% this time fine.

I am worried about it randomly stopping to update its blockchain though (i.e. the API calls work fine, it just doesn't get new blocks anymore, and I do not see errors in the log as to why).

[robby-d]

Actually, it's doing it again!

/sync returns:

{"status":"finished","blockChainHeight":205433,"syncPercentage":100,"syncedBlocks":205435,"error":null,"type":"from RPC calls","startTs":1395183596831,"endTs":1395183608691}

However, http://test.bitcore.io/ says the latest block is currently 205436!

I see no errors in the log file. I restart the insight server and I get:

insight server listening on port 3001 in development mode
Counting connected blocks. This could take some minutes
[historic_sync] Resuming sync from block:00000000a404e548713fa9ec9f0b2d8d59bf341162d0b8ca2df028960b9d4ce5
[historic_sync] Got 205437 blocks in current DB, out of 205436 block at bitcoind
[historic_sync] syncing from RPC (slow)
[historic_sync] Starting from:  00000000a404e548713fa9ec9f0b2d8d59bf341162d0b8ca2df028960b9d4ce5
[historic_sync] status: [100%]
Done Syncing { status: 'finished',
  blockChainHeight: 205436,
  syncPercentage: 100,
  syncedBlocks: 205438,
  syncTipHash: undefined,
  error: null,
  type: 'from RPC calls',
  startTs: 1395186243685,
  endTs: 1395186243687 }
[historic_sync] status: [100%]
{ '0': 'connecting to 127.0.0.1:18333' }
{ '0': 'connected to 127.0.0.1:18333' }
[p2p_sync] Connected to 1 peer

I can switch to using RPC and see if that fixes it (odd that it says 'from RPC calls above' ...maybe it is a file issue)...but this causes major issues when the site when insight randomly stops pulling new blocks. :/

[robby-d]

The odd thing is that on an initial startup, it syncs from the data files fine, but on subsequent startups, it uses RPC?

I am launching it via the following init script: https://github.com/xnova/counterpartyd_build/blob/develop/dist/linux/init/insight-testnet.conf.template

(where that !RUN_AS_USER! is actually replaced by a username when the init script is installed on the system)... it launches fine with this script.

I wish I could see why it is not doing a file-based sync on restart...getting no errors in the logs, just what I posted earlier....

[robby-d]

And here it is with the same problem on a different server:

$ curl http://127.0.0.1:3001/api/sync
{"status":"finished","blockChainHeight":205431,"syncPercentage":100,"syncedBlocks":206060,"error":null,"type":"from RPC calls","startTs":1395181731798,"endTs":1395181731800}

(out of sync by like 8+ blocks). Restarting insight on that system syncs it back up again properly. That system itself has had nothing else making requests to insight (it's a currently idle development box).

Any guidance here would be appreciated. This is the one big item holding back our product launch, unfortunately. :/

[robby-d]

Ok, last post for a while. :) I wanted to confirm it wasn't bitcoind that was slow. I ran the 'getinfo' command, which returns:

{
    "version" : 80600,
    "protocolversion" : 70001,
    "walletversion" : 60000,
    "balance" : 16.14943606,
    "blocks" : 205438,
    "timeoffset" : -18,
    "connections" : 9,
    "proxy" : "",
    "difficulty" : 1.00000000,
    "testnet" : true,
    "keypoololdest" : 1390234046,
    "keypoolsize" : 101,
    "paytxfee" : 0.00000000,
    "errors" : ""
}

Meanwhile, sync on the same box (hitting that bitcoind instance) is returning:

{"status":"finished","blockChainHeight":205437,"syncPercentage":100,"syncedBlocks":206066,"error":null,"type":"from RPC calls","startTs":1395188216085,"endTs":1395188216088}

I.e. 205438 (bitcoind) vs 205437 (insight), where 205438 came out ~14 minutes ago as of this writing. So unless I'm off, this definitely appears to be an issue with insight not properly syncing against bitcoind (and bitcoind looks current).

[robby-d]

As a temporary workaround, I made a crontab entry like:

*/2 * * * * killall -HUP /usr/bin/node

However this isn't optimal as I think it will probably kill/delay any API requests at the time, right?

And by the way, I have 3 separate Ubuntu 13.10 x64 servers with Insight installed (latest code), and it is doing this same thing on every single one.

[robby-d]

And lastly, I noticed that your own production platform is even showing this issue, it appears. Going to http://test.bitcore.io/api/sync shows the following (even after a shift-refresh):

{"status":"finished","blockChainHeight":205257,"syncPercentage":100,"syncedBlocks":205739,"error":null,"type":"from RPC calls","startTs":1395037653052,"endTs":1395037653060}

However, on your website (http://test.bitcore.io/), the blockchain height is listed as 205449 at the time of this writing.

[matiu]

Hi @xnova. Thanks for the report. It is pretty late here, can we meet tomorrow in irc to discuss this issue (#bitcore at freenode). 

Just some comments:

• historic synx switch to rpc when restarting and having more than 90% IIRC. Because in that case is faster
• historic sync only runs at server startup (or restart) once, to catch up to then blockchain while insight was offline. The percentage is related to that Process only. 
• the offset in the blockchain tip could be explained if insight did not received the block thru p2p. Could it be a communication issue with bicoind? Can you check insight block to see if the arrived?
  —
  Sent from Mailbox for iPhone

On Tue, Mar 18, 2014 at 9:31 PM, xnova [email protected] wrote:

Ok, last post for a while. :) I wanted to confirm it wasn't bitcoind that was slow. I ran the 'getinfo' command, which returns:

{
    "version" : 80600,
    "protocolversion" : 70001,
    "walletversion" : 60000,
    "balance" : 16.14943606,
    "blocks" : 205438,
    "timeoffset" : -18,
    "connections" : 9,
    "proxy" : "",
    "difficulty" : 1.00000000,
    "testnet" : true,
    "keypoololdest" : 1390234046,
    "keypoolsize" : 101,
    "paytxfee" : 0.00000000,
    "errors" : ""
}

Meanwhile, sync on the same box (hitting that bitcoind instance) is returning:

{"status":"finished","blockChainHeight":205437,"syncPercentage":100,"syncedBlocks":206066,"error":null,"type":"from RPC calls","startTs":1395188216085,"endTs":1395188216088}

I.e. 205438 (bitcoind) vs 205437 (insight), where 205438 came out ~14 minutes ago as of this writing. So unless I'm off, this definitely appears to be an issue with insight not properly syncing against bitcoind (and bitcoind looks current).

Reply to this email directly or view it on GitHub:
#17 (comment)

[matiu]

I mean insight log. —
Sent from Mailbox for iPhone

On Wed, Mar 19, 2014 at 4:56 AM, Matias Alejo Garcia [email protected]
wrote:

Hi @xnova. Thanks for the report. It is pretty late here, can we meet tomorrow in irc to discuss this issue (#bitcore at freenode). 
Just some comments:

• historic synx switch to rpc when restarting and having more than 90% IIRC. Because in that case is faster
• historic sync only runs at server startup (or restart) once, to catch up to then blockchain while insight was offline. The percentage is related to that Process only. 
• the offset in the blockchain tip could be explained if insight did not received the block thru p2p. Could it be a communication issue with bicoind? Can you check insight block to see if the arrived?
  —
  Sent from Mailbox for iPhone
  On Tue, Mar 18, 2014 at 9:31 PM, xnova [email protected] wrote:

  Ok, last post for a while. :) I wanted to confirm it wasn't bitcoind that was slow. I ran the 'getinfo' command, which returns:
  { "version" : 80600, "protocolversion" : 70001, "walletversion" : 60000, "balance" : 16.14943606, "blocks" : 205438, "timeoffset" : -18, "connections" : 9, "proxy" : "", "difficulty" : 1.00000000, "testnet" : true, "keypoololdest" : 1390234046, "keypoolsize" : 101, "paytxfee" : 0.00000000, "errors" : "" }
  Meanwhile, sync on the same box (hitting that bitcoind instance) is returning:
  {"status":"finished","blockChainHeight":205437,"syncPercentage":100,"syncedBlocks":206066,"error":null,"type":"from RPC calls","startTs":1395188216085,"endTs":1395188216088}
  I.e. 205438 (bitcoind) vs 205437 (insight), where 205438 came out ~14 minutes ago as of this writing. So unless I'm off, this definitely appears to be an issue with insight not properly syncing against bitcoind (and bitcoind looks current).

  Reply to this email directly or view it on GitHub:
  #17 (comment)

[matiu]

as discussed on #bitcore IRC, this seems to be problem with the /sync API only. the /sync, reporting the tip from the historical sync and not the actual insight block tip. Will correct that soon.

[matiu]

/api/status?q=getLastBlockHash

returns the last tip for bitcoind (from RPC) and the last tip from Insight.

This should resolve the issue. thanks.

[robby-d]

Hmm...I see I can get the block hash that way, and then make another call to /api/block/{hash} to get the "height" value. Is that the best way, given that there's no single step way to do it? Also, would you consider adding this last block hash and last block index to /api/sync (or at least changing the blockChainHeight parameter name maybe, to make it more clear)??

[robby-d]

And actually, /api/status?q=getInfo (getInfo) lists the last block index. That may be best.

[robby-d]

@matiu FYI, our new wallet, which uses insight-api for BTC balance/utxo info is released in beta on testnet:

https://www.counterparty.co/counterwallet-live-testnet/
https://testnet.counterwallet.co

Thanks a lot to you guys for making insight, and for the help earlier!

[matiu]

@xnova That is awesome! Great to know that Insight is used in your wallet. please let us know how it is working and keep us posted about your product. Good luck!

[matiu]

@xnova just created an account, beautiful UX! congratulations.

[robby-d]

@matiu awesome! glad you are liking the UI. Matt from our team is reaching out to you guys to see if we can do a case study/showcase with your team for our use of insight.... i.e. Bitpay technology getting some real cool real world use :)

[matiu]

@xnova jfyi .In the homepage of 'https://testnet.counterwallet.co/' I can't scroll down (using chrome here).

[abstream]

@robby-dermody what was your final solution for insight-api out of sync problem?
I experience the very same issue and my first thought was to regularly restart insight no matter of the last block hash. any help is appreciated.

Thanks.

[abstream]

@matiu /api/status?q=getLastBlockHash now return 2 values: syncTipHash , lastblockhash
shall i conclude that something is wrong if those values are different?

[robby-d]

@abstream in our case an update to the newest master fixed it, I believe. We might had had to resync/rebuild the db as well... It's been awhile and my memory is fuzzy on this

[abstream]

@robby-dermody , just to put info for others interested on the topic,
I got the code yesterday with this command:

git clone https://github.com/bitpay/insight-api && cd insight-api

so, I guess it is up to date. I managed to get rid off the issue by setting both p2p port and b_port to 9332 (using litecoin at the moment). Now insight syncs fine, does not disconnect, but i intend to check regulary(cron + email) /api/peer and act manually if needed.

[abstream]

Ok, i shouldn't celebrate so soon....after changing the ports it stopped to disconnect me but the sync still stops after 100% are reached. It would not update...it just considers that 100% are reached and one need to restart the api in order to resync... i cannot call util/sync.js while the api is running cause it is locked

[9cat]

I am still encounter this problem when version insight API v0.2.12