authgear / authgear-nft-indexer

AuthGear NFT Indexer

Prerequisite

0. Docker 20.10.17+

1. Install asdf

2. Run the following to install all dependencies in .tool-versions

asdf install

Environment Setup

1. Run the following to generate a config file

make setup

2. Edit authgear-nft-indexer.yaml for applicable configurations

Database setup

1. Start the db container

docker compose up -d

2. Apply database schema migrations:

   make sure the db container is running

   go run ./cmd/server database migrate up

To create new migration:

# go run ./cmd/authgear database migrate new <migration name>
go run ./cmd/indexer database migrate new add user table

Run everything

docker compose up -d

Then run the following command to start up the server

# in project root
make start

authgear-nft-indexer's Issues

Setup contract scanning microservice

• Setup database
  • Create nft_collection and nft_transfers tables
  • Create nft_owner view
• Setup local redis
• Setup api for alchemy
• Initialize gocron scheduler
  • Check db and enqueue collections for sync every 5 minues
• Initialize go-worker
  • Handle synchronizing nft transfer records to local db

Worker scheduling issues

• When the same collection was added twice, two jobs were scheduled. They would run in parallel, duplicating efforts.
• The job should be able to stop itself. It should be done by determining whether the synced block height is new enough.

Block large collection

Large collection involves many transfers which would incur more expensive storage cost.

Setup API server for indexer service

• Setup GIN server
  • contract address registration/deregistration
  • list all tokens under a collection, paginated
  • list available tokens under an address

Indexer API adjustment

As we will be adding a detail page as well as validation prior to adding collections

We shall address the following

• Add API for token listing
• Add token type and total supply to collection table
• Allow collection listing api to filter by contract id

Handle indexer support for ERC-1155 collections

• Add value to transaction and owner table
• Update indexer to support 1155
  • Handle token uniqueness with value

TODO: Update issue points

refs #2489

MVP support for NFT

• Drop support for snapshot
• Drop the indexer
• Keep the API server
• Replace the watch endpoint with the probe endpoint.
  • The probe endpoint blocks large NFT collection with getNFTsForCollection. If the response has page key, it implies the collection may be too large.
• Rewrite the nfts endpoint
  • It always read from the table eth_nft_owner of the database to return a response
  • It determines whether it needs to update eth_nft_owner before returning a response
    • It does so by storing nft_owner (blockchain, network, address, last_sync_at)
  • It takes owner_address and contract_ids.
  • It calls getNFTs for at most 5 times (configurable) to get the NFTs owned by a given address
  • It calls getAssetTransfers with to_address and the intersection of (contract_ids and the response of getNFTs) at most 5 times to upsert eth_nft_owner.

Rename `synced_block_height` to `from_block_height`

from_block_height means which block height should our worker start syncing from.
It MUST never be negative. So check the trigger to only update it when the new value is non-negative.
Just edit the initial schema in-place is fine. No need to write another migration to rename the column.

Update NFT server

As discussed offline,

We will update the server as follows

• Remove nft_owner table
• Make eth_nft_ownership an insert-only table

Flow:

• Calls GetOwnersForCollection to probe the collection
• Calls getNFTs to get the ownerships
• Calls getAssetTransfer to get the ownership detail
• If owns
  • Insert accordingly to the ownership table
• else
  • Insert a row with empty data to the ownership table
• Fetch ownership from table
• Append data and return if data is not empty

Change mapped service ports to something else to avoid clash with authgear-server

During development, we need to run both authgear-server and authgear-nft-indexer. Their docker-compose.yaml map to the same port on the host machine. Let's change the ports to something else.

Add owner scoping to indexer api server

As discussed with Louis offline, we think that it would be simpler to:

1. Remove collection deregistering api
2. Change collection register api to watch
   • No-op if collection already exists
   • Insert if it's a new collection
3. Find a way to scope contract address when inquiring owner nfts
4. Remove unnecessary api

Setup Github Actions