a16z / nft-analyst-starter-pack Goto Github PK

View Code? Open in Web Editor NEW

462.0 462.0 92.0 28.28 MB

Home Page: https://a16z.com/2022/03/18/nft-starter-pack-analyze-data-metadata-build-tools/

License: GNU Affero General Public License v3.0

Python 99.48% Dockerfile 0.52%

analytics data-science ethereum nfts python

nft-analyst-starter-pack's People

Contributors

Stargazers

Watchers

Forkers

gpalrepo trader-xyz nataliemeurer oranker adwilkinson genysys icartic pkiage problem-child appbootup goodvibesohmly linetoanywhere brianleect satyam-cyc ngmisl bitsofether bienpx224 mingfengkeith dlevine815 jcity witwiki guirkgumbo pynchmeister bolak chainleft josephmartinmoran itsdevbear 0xfacedefi shawn888 ers123 paulpierre franceshe drmichaelwang mseidlx alice3773 ilerik ndehouche ratkoj rthatha bbenzikry kandy22 kpnkpnkpn daldiv turingplus ivapx wakamex woods1060 jinnzy automatedtradingcompany rootslashbin bitpack-me machikacryps 0xmgg jdev214 samspzn damian87x orderwithinchaos ferasalsaab unbanksytv infininight 0xcracka jarihte bhou831 kickerbtc rugzy-eth decentered furmanlukasz mikimaus78 wtiger0605 jithinraj cian0 sarvex takuya-wakiyama mektigboy dattgoswami wbnns forkoooor mishanefedov ethcanal jdot-dev ssghost ikigailabseth marcusdiaz linxux aprudnik andrewfowl jfparra taz-rb

nft-analyst-starter-pack's Issues

Coingecko ETH Price API delay

We might want to add some exception handling to the jobs/update_eth_prices.py because the entire job fails if the coingecko API isn't up-to-date with ETH pricing data.

Today's September 11th. Coingecko has prices for the 9th, but not the 10th or 11th. This throws an error at line 48 of update_eth_prices.py. An example return on a failure day:

{"id":"ethereum","symbol":"eth","name":"Ethereum","localization":{"en":"Ethereum","de":"Ethereum","es":"Ethereum","fr":"Ethereum","it":"Ethereum","pl":"Ethereum","ro":"Ethereum","hu":"Ethereum","nl":"Ethereum","pt":"Ethereum","sv":"Ethereum","vi":"Ethereum","tr":"Ethereum","ru":"Эфириум","ja":"イーサリアム","zh":"以太坊","zh-tw":"以太幣","ko":"이더리움","ar":"يثريوم","th":"Ethereum","id":"Ethereum","cs":"Ethereum","da":"Ethereum","el":"Ethereum","hi":"Ethereum","no":"Ethereum","sk":"Ethereum","uk":"Ethereum","he":"Ethereum","fi":"Ethereum","bg":"Ethereum","hr":"Ethereum","lt":"Ethereum","sl":"Ethereum"},"image":{"thumb":"https://assets.coingecko.com/coins/images/279/thumb/ethereum.png?1595348880","small":"https://assets.coingecko.com/coins/images/279/small/ethereum.png?1595348880"}}

No market_data key. Any suggestions for how to handle?

Using the code for Adidas NFT Collection - getting the below error

I was trying to use this code for Adidas NFT Collection (0x28472a58A490c5e09A238847F66A68a47cC76f0f)

I get the below error

Process started for contract address: 0x28472a58A490c5e09A238847F66A68a47cC76f0f
Exporting token transfers...
Traceback (most recent call last):
File "export_data.py", line 164, in
export_data()
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "export_data.py", line 88, in export_data
export_token_transfers(
File "/Users/viper/Documents/OneDrive/TellerFinance/nft-analyst-starter-pack/jobs/export_token_transfers.py", line 27, in export_token_transfers
token_transfers = pd.read_csv(output)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
return _read(filepath_or_buffer, kwds)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 575, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 933, in init
self._engine = self._make_engine(f, self.engine)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1231, in _make_engine
return mapping[engine](f, **self.options)
File "/Users/viper/opt/anaconda3/lib/python3.8/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 75, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 551, in pandas._libs.parsers.TextReader.cinit
pandas.errors.EmptyDataError: No columns to parse from file`

Key error when crawling cryptopunks data

I am trying to use this on 0xb47e3cd837dDF8e4c57F05d70Ab865de6e193BBB to crawl cryptopunks data
However a key error was thrown when running Updating ETH prices...

Process started for contract address: 0xb47e3cd837dDF8e4c57F05d70Ab865de6e193BBB
Exporting token transfers...
Exporting logs...
Updating block-to-date mapping...
Updating ETH prices...
Traceback (most recent call last):
  File "/Users/william/nft-analyst-starter-pack/export_data.py", line 164, in <module>
    export_data()
  File "/Users/william/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/william/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/william/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/william/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/william/nft-analyst-starter-pack/export_data.py", line 131, in export_data
    generate_sales_output(
  File "/Users/william/nft-analyst-starter-pack/core/generate_sales_output.py", line 41, in generate_sales_output
    transfers_df["date"] = transfers_df["block_number"].apply(
  File "/Users/william/opt/anaconda3/lib/python3.9/site-packages/pandas/core/series.py", line 4433, in apply
    return SeriesApply(self, func, convert_dtype, args, kwargs).apply()
  File "/Users/william/opt/anaconda3/lib/python3.9/site-packages/pandas/core/apply.py", line 1082, in apply
    return self.apply_standard()
  File "/Users/william/opt/anaconda3/lib/python3.9/site-packages/pandas/core/apply.py", line 1137, in apply_standard
    mapped = lib.map_infer(
  File "pandas/_libs/lib.pyx", line 2870, in pandas._libs.lib.map_infer
  File "/Users/william/nft-analyst-starter-pack/core/generate_sales_output.py", line 42, in <lambda>
    lambda x: date_blocks_df.iloc[date_blocks_df.index.get_loc(x)]["date"]
  File "/Users/william/opt/anaconda3/lib/python3.9/site-packages/pandas/core/indexes/interval.py", line 637, in get_loc
    raise KeyError(key)
KeyError: 3919706```

It works only with eth-mainnet

Alchemy works with other chains, but here is only working with eth-mainnet.
In some places in the code the url is set to eth-mainnnet:
alchemy_url = "https://eth-mainnet.g.alchemy.com/{alchemy_api_key}/v1

Results contain duplicate data

Thank you for your open source project, I found duplicate data in the results during use.

code:

nft-analyst-starter-pack/core/generate_sales_output.py

Line 98 in e14844a

sales_df = logs_df.merge(transfers_df, on="transaction_hash", how="left")

The merge function will cause duplicate data.

test

import pandas as pd
df1 = pd.DataFrame({"name":["kate","sally"],
                    "age":[25,285]})
df2 = pd.DataFrame({"name":["kate","herz","sally",'sally'],
                     "score":[70,60,11,11], "age":[23,41,285,44]})
print(pd.merge(df1,df2,on="name", how='left'))

output

    name  age_x  score  age_y
0   kate     25     70     23
1  sally    285     11    285
2  sally    285     11     44

export_data.py throws errors on the imports. Cannot run the script.

File "C:\Users\***NAME***\nft-analyst-starter-pack\export_data.py", line 14, in <module>
    import click

Does this for ethereumetl, numpy, pandas, ethereumetl.service.eth_service, and web3. I am using Python 3.9.12. I get that this is an error that I have created for myself, but do you know of any fixes off the top of your head?

KeyError: 'market_data'

I'm geting this error while trying to get data from foundation nft contract

Updating block-to-date mapping...
Updating ETH prices...
{'id': 'ethereum', 'symbol': 'eth', 'name': 'Ethereum', 'localization': {'en': 'Ethereum', 'de': 'Ethereum', 'es': 'Ethereum', 'fr': 'Ethereum', 'it': 'Ethereum', 'pl': 'Ethereum', 'ro': 'Ethereum', 'hu': 'Ethereum', 'nl': 'Ethereum', 'pt': 'Ethereum', 'sv': 'Ethereum', 'vi': 'Ethereum', 'tr': 'Ethereum', 'ru': 'Эфириум', 'ja': 'イーサリアム', 'zh': '以太坊', 'zh-tw': '以太幣', 'ko': '이더리움', 'ar': 'يثريوم', 'th': 'Ethereum', 'id': 'Ethereum', 'cs': 'Ethereum', 'da': 'Ethereum', 'el': 'Ethereum', 'hi': 'Ethereum', 'no': 'Ethereum', 'sk': 'Ethereum', 'uk': 'Ethereum', 'he': 'Ethereum', 'fi': 'Ethereum', 'bg': 'Ethereum', 'hr': 'Ethereum', 'lt': 'Ethereum', 'sl': 'Ethereum'}, 'image': {'thumb': 'https://assets.coingecko.com/coins/images/279/thumb/ethereum.png?1595348880', 'small': 'https://assets.coingecko.com/coins/images/279/small/ethereum.png?1595348880'}} 777
Traceback (most recent call last):
  File "/Users/luki/Documents/GitHub/nft-analyst-starter-pack/export_data.py", line 225, in <module>
    export_data()
  File "/Users/luki/opt/anaconda3/envs/dask-env/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/luki/opt/anaconda3/envs/dask-env/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/luki/opt/anaconda3/envs/dask-env/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/luki/opt/anaconda3/envs/dask-env/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/luki/Documents/GitHub/nft-analyst-starter-pack/export_data.py", line 173, in export_data
    update_eth_prices(filename=eth_prices_csv)
  File "/Users/luki/Documents/GitHub/nft-analyst-starter-pack/jobs/update_eth_prices.py", line 48, in update_eth_prices
    price_of_eth = j["market_data"]["current_price"]["usd"]
KeyError: 'market_data'

python export_data.py --alchemy-api-key APIKEY --contract-address 0x7498aCde901d950B823DDBe302E1B2daD41Cc308

Support for NFT events?

A NFT contract I was trying to pull data for was mysteriously missing mint transfers from the 0 addr. It turned out that the mints were triggered from a separate minting module contract. This minting module contract called the mint function in the actual NFT contract after a series of checks.

The actual NFT contract had transactions only for transfers that happened after minting. This was consistent with the data I pulled and what the transactions tab was showing on Etherscan. However, the events tab on the token showed all the transfers from the 0 addr and I was able to query those using checkthechain.

The minting module contract had transactions for the mints but the nft-analyst-starter-pack was unable to detect erc721 transfers (see traceback at the bottom).

Here's the minting module and the nft contract I'm referring to.

Questions:

Am I right in understanding that this utility only pulls transfers that show up as transactions?
Might there be a way to pull transfer events omitted from the contracts we refer to in order to ensure high coverage? My understanding is this smart contract pattern is somewhat common (mints coming from minting module, NFT contract separate).

nft-analyst-starter-pack git:(main) ✗ poetry run python export_data.py -a $ALCHEMY_API_KEY -c 0xe200dd50001311e65282d19849e9a93c5b37f1ce
Process started for contract address: 0xe200dD50001311e65282d19849e9A93C5B37F1ce
Checking update logs for most recent block...
No existing data. Contract 0xe200dD50001311e65282d19849e9A93C5B37F1ce appears to have been deployed at block 15391107
Exporting token transfers...
No ERC-721 transfers were identified. Therefore, searching for and extracting any ERC-1155 transfers.
Exporting 1155 transfers...
Exporting logs...
Updating block-to-date mapping...
Updating ETH prices...
Traceback (most recent call last):
  File "/Users/vintrocode/work/passage/pascal-the-chain/nft-analyst-starter-pack/export_data.py", line 225, in <module>
    export_data()
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/vintrocode/work/passage/pascal-the-chain/nft-analyst-starter-pack/export_data.py", line 176, in export_data
    generate_sales_output(
  File "/Users/vintrocode/work/passage/pascal-the-chain/nft-analyst-starter-pack/core/generate_sales_output.py", line 83, in generate_sales_output
    logs_df = pd.read_csv(logs_file)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 575, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 933, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1235, in _make_engine
    return mapping[engine](f, **self.options)
  File "/Users/vintrocode/Library/Caches/pypoetry/virtualenvs/nft-analyst-starter-pack-W3ad-lpQ-py3.9/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 75, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 551, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file

429 Client Error: Too Many Requests for url

I continue to get this error and the program ultimately ends before getting to any metadata creations. I only need the metadata file to get rarity calculations; however, even if I comment out the transaction and sales functions within export_data.py and just start at get_metadata_for_collection it will generate a metadata csv without any rarity calculations

429 Too Many Requests Error

When running export.py, I am now getting an error in the form of 'HTTPError: 429 Client Error: too Many Requests for url: https://eth-mainnet.alchemyapi.io/v2/[my api key]'

I've only started getting this error over the last couple of days. Prior to this, export.py was working well. I'm wondering if others are getting this error, and if you have any suggestions. Thanks.

X2Y2 support

Thanks for the repo here - has been helpful!

As near as I can tell this covers sales only on Opensea and Looksrare, wondering if there are plans to add x2y2? There has been decent amounts of traffic there lately due to their incentives.

Parsing errors

Hi!

First of all: thanks for creating this repository, it has saved me a lot of time!

Second: here is a couple of transactions that are being parsed wrong by the system (I suggest comparing the script output with Etherscan):

0x4d47350cc0912e292fb399c84d238156a1097c3589ca2bc47acb829989b97739
0x9058ad0eee3a29b8f407216a6c36d86ab6506876f663e1bcf9333176517be28a
0x47c6897e3de42585403e8ecd198c4211be05375f8cdc1fa636af6a5cab436611
0x4d47350cc0912e292fb399c84d238156a1097c3589ca2bc47acb829989b97739
0x0c67d461758c71799d52ea643190a4f43a091c66af5eba19f0aecccee30e0247

Best,
J

a16z / nft-analyst-starter-pack Goto Github PK

nft-analyst-starter-pack's People

Contributors

Stargazers

Watchers

Forkers

nft-analyst-starter-pack's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs