joincolony / block-ingestor Goto Github PK

Implement improvements pointed out in the review of #60

And the conclusions that were drawn in the subsequent call: https://www.notion.so/colony/Permissions-Events-26-04-23-ea5f7fe0b34c42289056afd95fcbc1f5

The most relevant here being refactoring the logic behind role events being handled one at a time vs. (currently) on a tx basis (all events in the transaction)

Note: This was extracted into this issue in the interest of time, as to be able to launch faster

Following a conversation with @bogdan-1337, this is the easiest solution to persist the stats between different ingestor builds

Currently, block-ingestor goes through all events starting from block number 1 to fetch and write past extensions/actions related data. This can (and should) be improved by starting the scanning at the last processed block number before the block-ingestor went down.

As in the title. Presumably the same would happen if OneTxPayment wasn't installed.

This might be a good opportunity to ensure all code related to getting clients uses ~utils/clients - there's a function for getting a Staked Expenditure client already and we'd need one for OneTxPayment.

Upon start, block-ingestor should go over all the past events and write the following to the database:

• Extension version information (by traversing through ExtensionAddedToNetwork events)
• Extension installation information (by traversing through ExtensionInstalled/ExtensionUninstalled events)
• Extension initialisation sstatus (by traversing through ExtensionInitialised events)

Currently a failed get block call will result in ingestor throwing an error and restarting which takes about 2 minutes. During that time no data will be processed and stored in the database, potentially resulting in a considerable delay in the UI. This issue is to implement a retry strategy so that it retries the get block call a number of times before throwing an error.

Currently, when updating domain metadata upon finalizing a motion, we overwrite the entire metadata state with the pending metadata state at the time the motion was created.

This will overwrite state changes that happened in the mean time.

This can be fixed reasonably straightforwardly (I think). See https://github.com/JoinColony/block-ingestor/pull/74/files/a90322d71d54f52a1707e207cdef907adc60e33c#r1195211845

Specification

Story

In the UI, we display verified checkmarks for users which are added as verified.
We need to realign this according to the verified members rework.

Description

This should be done by changing the isVerified flag on the ColonyContributor model.

Implementation

This issue is complete when...

• The isVerified flag and checkmarks reflect the state of verified members after realign

Description

New Colony network contract releases and extension releases are not being picked up by the block-ingestor.

This has been noticed in the past, but has most recently been noticed again as there was a new Colony Network deployment this morning to version 14.

The versions for each are as follows:

• ColonyNetwork:
  • Version in db: 13
  • Latest version: 14
• VotingReputation Extension:
  • Version in db: 9
  • Latest version: 10
• OneTxPayment Extension:
  • Version in db: 5
  • Latest version: 6

Note: The Colony network version has been manually updated in the database now to version 14.

I think it will be useful to write the latest version for colonies (and extensions) to the stats file as well.
This will help with checks once we get in production.

Originally posted by @rdig in #4 (comment)

Specification

Story

The block ingestor must be able to parse the new event which corresponds to adding verified members via permissions.
Then it needs to call the graphql mutation for adding verified members.

Description

The client will send out a Metadata delta event where they will pass through the operation object and the block ingestor needs to parse it and handle it.

Implementation

Create a new handler in the event handlers folder and handle it in eventProcessor.
We need to parse the passed in string, so we know which users are being added to which colony.

This issue is complete when...

• Block ingestor can successfully handle the event and discern it being an Add event not anything else
• The gql mutation is called and the database table is updated

Currently we don't write the colony data to the database, meaning we don't have access to all the colonies created before block-ingestor has started and we have to rely on frontend mutating the db directly (which we want to go away from).

This issue can be split in two parts:

• The trackColonies function should be extended to not only set up the colony specific listeners but also write the colony to the database. This is to ensure we store info on all previously created colonies
• Writing colony info to the db when handling ColonyAdded event

What is a reorg?

https://river.com/learn/terms/r/reorganization
https://cointelegraph.com/explained/what-is-chain-reorganization-in-blockchain-technology
https://www.alchemy.com/overviews/what-is-a-reorg

How to handle

Here are my notes from a call I had with Jakub, Raul and Alex:

when listening to blocks, for each block that comes in, we need to confirm that it's parent hash is the previous block we processed, else it's a reorg.

If it's a reorg, we need to go back the last block we saw that is consistent with the reorg, and re-sync from that block

We need to keep a list of the last 500 blocks we have processed, each block containing:

• block number
• block hash

-- We also need to think about undoing state changes in the db for blocks that got reorged (and which may, thus, no longer exist on chain)

Recommendations

Speak to Alex to remind yourself of the details before starting work on this.

Investigate and setup GraphQL codegen solution, preferably similar to the one in CDapp.

Currently, the ExtensionInitialised listener doesn't get removed when the extension is uninstalled. We should remove it. Follow pattern from #36

I wonder if we could optimize the ingestor somewhat if we were to store all known instances of the colony client.

I see us re-instantiating the client for a lot of tracked events.

But I actually don't know what this impact would be real-life.

Originally posted by @rdig in #22 (comment)

This issue is to investigate and work on or close the ticket.

Similarly to #38, we want to cache token clients in a bid to improve performance.

Currently, if the ingestor looses connection to the GraphQL api, more specifically the stats entry, it will reset from the first chain block, and more problematic, overwrite the stats entry.

This will render the block ingestor effectively useless until it catches up to the current block, which, starting from block 1 will take a while (order of weeks, multiple)

A common scenario where this can happen is during the short lived window when an API expire, and before the renew lambda process gets triggered.

The proposed solution for this is to prevent the above behavior by having the ingestor stop it's block fallback on API connection loss and just keep retrying until the connection is restored. That way, we avoid overwriting valid production data will fallbacks

Specification

Story

The block ingestor must be able to parse the new event which corresponds to removing verified members via permissions.
Then it needs to call the graphql mutation for removing verified members.

Description

The client will send out a Metadata delta event where they will pass through the pseudocode language a la REMOVE {userAddress} VERIFIED {colonyAddress} and the block ingestor needs to parse it and handle it.

Implementation

Create a new handler in the event handlers folder and handle it in eventProcessor.
We need to parse the passed in string, so we know which users are being removed from which colony's verified members.
After that, call the removeVerifiedMembers mutation.

This issue is complete when...

• Block ingestor can successfully handle the event and discern it being a Remove event not anything else
• The gql mutation is called and the database entry is modified

Similarly to #38, we want to cache extension clients (Voting Reputation is the only one for now) in a bid to improve performance. This needs to be branched off Will's motion work.

This is commented out by Raul in #60 since it interfered with the QA launch for whatever reason. The issue is to debug it locally while connected to QA and fix so that the tracking can be re-enabled

Using the list version of ColonyFundsClaimed query could be improved here. This will list all ColonyFundsClaims in the db, and only apply filtering for colony and token once the query returns. If we have more than 100 items in the db (the default limit), we may not receive the results we're expecting.

We should be able to query the colony directly and get claims from there, since they're connected in the schema.

Specification

Story

The block ingestor must be able to parse the new event which corresponds to removing verified members via motions.
When it's finalized it needs to apply the result (or not apply it if it failed).

Description

The client will send out a Metadata delta motion event where they will pass through the pseudocode language a la REMOVE {userAddress} VERIFIED {colonyAddress} and the block ingestor needs to parse it and handle it.

Implementation

Create a new handler in the motions/motionCreated folder and handle it in motionCreated.ts file.
We need to parse the passed in string, so we know which user is being removed from which colony's verified members.

This issue is complete when...

• Block ingestor can successfully handle the motion event and discern it being Remove event not anything else
• Block ingestor successfully creates a motion
• If the motion passes, we remove the verified members via the gql mutation
• If the motion fails the members don't get removed

Handle the following events to extract required permissions:

• ColonyRoleSet(indexed address,indexed uint256,indexed uint8,bool) -- old version of event
• ColonyRoleSet(address,indexed address,indexed uint256,indexed uint8,bool) -- current version of event
• RecoveryRoleSet(indexed address,bool) -- recovery role

Note to extract domain, and block at which point this permission was set.

A good idea is to look up the way we used to fetch permissions in colonyJS: https://github.com/JoinColony/colonyJS/blob/deprecated/4.x.x-branch-still-in-use/src/helpers.ts#L194-L255

Several issues with potential solutions were raised during the ingestor call on Friday, May 2nd, including:

• Changing the order of setting up listeners
• Using 2 queues to ensure no events are missed during the tracking stage
• Retrieving known colonies from the DB instead of the chain

The current setup relies on the developer having a linting extension enabled to highlight any issues during development. There are no pre-commit/pre-push hooks that would run linting automatically.

The issue is to investigate the existing CDapp linting setup and port the relevant parts to block-ingestor.

Specification

Story

The block ingestor must be able to parse the new event which corresponds to adding verified members via motions.
When it's finalized it needs to apply the result (or not apply it if it failed).

Description

The client will send out a Metadata delta motion event where they will pass through the pseudocode language a la ADD {userAddress} VERIFIED {colonyAddress} and the block ingestor needs to parse it and handle it.

Implementation

Create a new handler in the motions/motionCreated folder and handle it in motionCreated.ts file.
We need to parse the passed in string, so we know which user is being added to which colony.

This issue is complete when...

• Block ingestor can successfully handle the motion event and discern it being an Add event not anything else
• Block ingestor successfully creates a motion
• If the motion passes, we add the verified members via the gql mutation
• If the motion fails the members don't get added