Autoretrieve is a standalone gateway server / Bitswap provider that bridges Filecoin Graphsync content over to Bitswap clients.
When a Bitswap client requests data from Autoretrieve, Autoretrieve asks either Estuary or an STI indexer which Filecoin service providers are hosting it, starts a retrieval deal with a selected SP, and streams the incoming blocks to the Bitswap client.
Autoretrieve uses Docker with Buildkit for build caching. Docker rebuilds are quite fast, and it is usable for local development. Check the docker-compose documentation for more help.
$ DOCKER_BUILDKIT=1 docker-compose up
You may optionally set FULLNODE_API_INFO
to a custom fullnode's WebSocket
address. The default is FULLNODE_API_INFO=wss://api.chain.love
.
By default, config files and cache are stored at ~/.autoretrieve
. When using
docker-compose, a binding is created to this directory. This location can be
configured by setting AUTORETRIEVE_DATA_DIR
.
Internally, the Docker volume's path on the image is /root/.autoretrieve
. Keep
this in mind when using the Docker image directly.
Some CLI flags and corresponding environment variables are available for basic configuration.
For more advanced configuration, config.yaml
may be used. It lives in the autoretrieve data directory, and will be automatically generated by running autoretrieve. It may also be manually generated using the gen-config
subcommand.
Configurations are applied in the following order, from least to most important:
- YAML config
- Environment variables
- CLI flags
advertise-endpoint-url: # leave blank to disable, example https://api.estuary.tech/autoretrieve/heartbeat (must be registered)
advertise-endpoint-token: # leave blank to disable
lookup-endpoint-type: indexer # indexer | estuary
lookup-endpoint-url: https://cid.contact # for estuary endpoint-type: https://api.estuary.tech/retrieval-candidates
max-bitswap-workers: 1
routing-table-type: dht
prune-threshold: 1GiB # 1000000000, 1 GB, etc. Uses go-humanize for parsing. Table of valid byte sizes can be found here: https://github.com/dustin/go-humanize/blob/v1.0.0/bytes.go#L34-L62
pin-duration: 1h # 1h30m, etc.
log-resource-manager: false
log-retrieval-stats: false
disable-retrieval: false
cid-blacklist:
- QmCID01234
- QmCID56789
- QmCIDabcde
miner-blacklist:
- f01234
- f05678
miner-whitelist:
- f01234
default-miner-config:
retrieval-timeout: 1m
max-concurrent-retrievals: 1
miner-configs:
f01234:
retrieval-timeout: 2m30s
max-concurrent-retrievals: 2
f05678:
max-concurrent-retrievals: 10
$ autoretrieve --help
NAME:
autoretrieve - A new cli application
USAGE:
autoretrieve [global options] command [command options] [arguments...]
COMMANDS:
gen-config Generate a new config with default values
print-config Print detected config values as autoretrieve sees them
check-cid Takes a CID argument and tries walking the DAG using the local blockstore
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--data-dir value [$AUTORETRIEVE_DATA_DIR]
--lookup-endpoint-url value Indexer or Estuary endpoint to get retrieval candidates from [$AUTORETRIEVE_LOOKUP_ENDPOINT_URL]
--lookup-endpoint-type value Type of endpoint for finding data (valid values are "estuary" and "indexer") [$AUTORETRIEVE_LOOKUP_ENDPOINT_TYPE]
--disable-retrieval Whether to disable the retriever module, for testing provider only (default: false) [$AUTORETRIEVE_DISABLE_RETRIEVAL]
--routing-table-type value [dht|fullrt|disabled] [$AUTORETRIEVE_ROUTING_TABLE_TYPE]
--log-resource-manager Whether to present output about the current state of the libp2p resource manager (default: false) [$AUTORETRIEVE_LOG_RESOURCE_MANAGER]
--log-retrievals Whether to present periodic output about the progress of retrievals (default: false) [$AUTORETRIEVE_LOG_RETRIEVALS]
--help, -h show help (default: false)