dfuse-io / dfuse-eosio Goto Github PK

dfuse for EOSIO

License: Apache License 2.0

Go 68.63% Shell 1.19% JavaScript 0.11% HTML 0.29% CSS 0.04% TypeScript 29.40% Emacs Lisp 0.03% Dockerfile 0.08% C++ 0.23%

eosio blockchain data dfuse streaming-search

dfuse-eosio's Introduction

EOSIO on StreamingFast

All dfuse.io services for EOSIO, running from your laptop or from a container, released as a single statically linked binary: dfuseeos.

See the general dfuse repository for other blockchain protocols implementations.

Getting started

If it's the first time you boot a nodeos node, please review https://developers.eos.io/welcome/latest/tutorials/bios-boot-sequence and make sure you get a grasp of what this blockchain node is capable.

The default settings of dfuseeos allow you to quickly bootstrap a working development chain by also managing the block producing node for you.

Requirements

Operating System

Linux or macOS (no Windows support for now, should work through WSL2 (untested))

dfuse Instrumented nodeos (deep-mind)

See DEPENDENCIES.md for instructions on how to get an instrumented nodeos binary.

Installing

From a pre-built release

Download a tarball from the GitHub Releases Tab.
Put the binary dfuseeos in your PATH.

From source

Build requirements:

Git
Go 1.14 or higher (installation)
yarn 1.15 or higher (installation)

./scripts/build.sh

This will install the binary in your $GOPATH/bin folder (normally $HOME/go/bin). Make sure this folder is in your PATH env variable. If it's missing, take a look at TROUBLESHOOTING.md.

Note -- If you're getting yarn dependency warnings while running the yarn install && yarn build commands above, you can normally safely ignore those and move forward with the installation. If you're getting an error while installing and/or compiling, see TROUBLESHOOTING.md.

Creating a new local chain with `dfuseeos`

1. Initialize

Initialize a few configuration files in your working directory (dfuse.yaml, mindreader/config.ini, ...)

dfuseeos init

Answer y (yes) when being prompted for Do you want dfuse for EOSIO to run a producing node for you?. If you intend is to sync an existing chain, follow Syncing an existing chain with dfuseeos instead.

2. Boot

Optionally, you can also copy over a boot sequence to have dfuse bootstraps your chain with accounts + system contracts to have a chain ready for development in a matter of seconds:

wget -O bootseq.yaml https://raw.githubusercontent.com/dfuse-io/dfuse-eosio/develop/devel/standard/bootseq.yaml

When you're ready, boot your instance with:

dfuseeos start

A successful start will list the launching applications as well as the graphical interfaces with their relevant links:

Dashboard:        http://localhost:8081

Explorer & APIs:  http://localhost:8080
GraphiQL:         http://localhost:8080/graphiql

In this mode, two nodeos instances will now be running on your machine, a block producer node and a mindreader node, and the dfuse services should be ready in a couple seconds.

Syncing an existing chain with `dfuseeos`

If you chose to sync to an existing chain, only the mindreader node will launch. It may take a while for the initial sync depending on the size of the chain and the services may generate various error logs until it catches up (more options for quickly syncing with an existing chain will be proposed in upcoming releases).

See Syncing a chain partially
See the following issue about the complexity of syncing a large chain

You should also take a look at our Docs:

Filtering

See Filtering

Overview - Repository Map

The glue:

The dfuseeos binary.
The launcher which starts all the internal services

The EOSIO-specific services:

abicodec: ABI encoding and decoding service
statedb: The dfuse State database for EOSIO, with all tables at any block height
kvdb-loader: Service that loads data into the kvdb storage
dashboard: Server and UI for the dfuse for EOSIO dashboard.
eosq: The famous https://eosq.app block explorer
eosws: The REST, Websocket service, push guarantee, chain pass-through service.

dfuse Products's EOSIO-specific hooks and plugins:

search plugin, object mappers, EOSIO-specific indexer, results mapper (along with the search client.
dgraphql resolvers, with all data schemas for EOSIO
blockmeta plugin, for EOS-specific kvdb bridge.

Logging

See Logging

Troubleshooting

See Troubleshooting

Contributing

Please read CONTRIBUTING.md for details on our Code of Conduct & processes for submitting pull requests, and CONVENTIONS.md for our coding conventions.

License

Apache 2.0

References

dfuse Docs
dfuse on Telegram - Community & Team Support

dfuse-eosio's People

Contributors

Stargazers

Watchers

dfuse-eosio's Issues

Error starting nodeos on Dfuse docker container - Post \"http://:8888/v1/chain/get_info\": dial tcp :8888: connect: connection refused"

We are running the following docker container

FROM ubuntu:18.04
RUN apt-get update \
    && apt-get install -y --no-install-recommends wget ca-certificates software-properties-common \
    && wget https://github.com/EOSIO/eosio.cdt/releases/download/v1.7.0/eosio.cdt_1.7.0-1-ubuntu-18.04_amd64.deb \
    && apt install -y ./eosio.cdt_1.7.0-1-ubuntu-18.04_amd64.deb \
    && rm eosio.cdt_1.7.0-1-ubuntu-18.04_amd64.deb \
    && wget https://github.com/dfuse-io/eos/releases/download/v2.0.5-dm-12.0/eosio_2.0.5-dm.12.0-1-ubuntu-18.04_amd64.deb \
    && apt install -y ./eosio_2.0.5-dm.12.0-1-ubuntu-18.04_amd64.deb \
    && rm eosio_2.0.5-dm.12.0-1-ubuntu-18.04_amd64.deb \
    && wget https://github.com/dfuse-io/dfuse-eosio/releases/download/v0.1.0-beta3/dfuse-eosio_0.1.0-beta3_linux_x86_64.tar.gz \
    && tar -xzvf dfuse-eosio_0.1.0-beta3_linux_x86_64.tar.gz \
    && rm dfuse-eosio_0.1.0-beta3_linux_x86_64.tar.gz
RUN apt-get install -y curl \
    && mv dfuseeos /usr/local/bin/dfuseeos \
    && apt-get remove -y wget ca-certificates \
    && apt-get autoremove -y \
    && rm -rf /var/lib/apt/lists/* \
    && cd /usr/opt/eosio.cdt/1.7.0/bin
RUN cd / \
    # "y" will initialize to run a producing node, "n" will connect to existing node
    && (echo "y" && cat) | dfuseeos init
RUN sed -i 's/127.0.0.1:8888/0.0.0.0:8888/g' producer/config.ini \
    && sed -i '9i plugin = eosio::history_plugin' producer/config.ini \
    && sed -i '10i plugin = eosio::history_api_plugin' producer/config.ini \
    && sed -i '11i filter-on=*' producer/config.ini \
    && sed -i '12i filter-out=eosio:onblock:' producer/config.ini

We then run the container with

docker run -p 8888:8888 -p 8081:8081 -p 8080:8080 -it dfuse dfuseeos start

Which shows the following error

"http://:8888/v1/chain/get_info: Post \"http://:8888/v1/chain/get_info\": dial tcp :8888: connect: connection refused"

System: Fresh Ubuntu 18.04 VM running through VirtualBox on Mac host system

Full verbose log file attached
dfuse.log

address all races detected by by go race detector

One of these may be related to #31

Starting dfuse for EOSIO with config file './dfuse.yaml'
Launching applications: abicodec,apiproxy,blockmeta,dashboard,dgraphql,eosq,eosws,fluxdb,merger,mindreader,node-manager,relayer,search-archive,search-indexer,search-live,search-router,trxdb-loader
==================
WARNING: DATA RACE
Write at 0x00c0004815d0 by goroutine 110:
  github.com/grpc-ecosystem/go-grpc-prometheus.WithHistogramBuckets.func1()
      /home/stepd/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/metric_options.go:32 +0x6c
  github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).EnableHandlingTimeHistogram()
      /home/stepd/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:65 +0x91
  github.com/grpc-ecosystem/go-grpc-prometheus.EnableHandlingTimeHistogram()
      /home/stepd/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server.go:46 +0x6c
  github.com/dfuse-io/dgrpc.NewServer()
      /home/stepd/go/pkg/mod/github.com/dfuse-io/[email protected]/server.go:141 +0xb52
  github.com/dfuse-io/dfuse-eosio/dashboard.(*server).Launch()
      /home/stepd/repos/dfuse-eosio/dashboard/server.go:75 +0x1c4
  github.com/dfuse-io/dfuse-eosio/dashboard.(*App).Run.func1()
      /home/stepd/repos/dfuse-eosio/dashboard/app.go:66 +0x3c
...

(see more attached)

dashboard's port 8080 is hard-coded in the TS code

We can now change the listening port for dashboard's grpc and http (like any other service), but the served up code defaults to localhost:8080/api when connecting back.

Public key in transaction search is not correct

The pub_keys value return in transaction queries GET /v0/search/transactions is not correct. We do not know what the keys shown are.

Walkthrough of problem
We bootstrap our blockchain with several transactions in bootstrap-accounts.js where we specify the eosio.token transfer arguments, and the private key to sign with. e.g.

  await transferTokens("gov", "thenewfork", 3000, "", settings.eosio.accounts.yvo.pkey);
  await transferTokens("thenewfork", "jack", 600, "", settings.eosio.accounts.jack.pkey);
  await transferTokens("thenewfork", "kirsten", 600, "", settings.eosio.accounts.jack.pkey);
  await transferTokens("thenewfork", "matej", 600, "", settings.eosio.accounts.jack.pkey);

You can see the pub/priv keys in settings.js

We search for transactions in our React app using the GET /v0/search/transactions endpoint using the js SDK

const query = "(auth:"+state.accountName+" OR receiver:"+state.accountName+")"
let transactionRes = await eosio.dfuseClient.searchTransactions(query);

and then print out the public key

const publicKey = trx.pub_keys[0];
const blockNum = trx.execution_trace.action_traces[0].block_num;
let keyRes;
console.log(trx.execution_trace.action_traces[0].act.name, trx.execution_trace.action_traces[0].act.data, blockNum)
console.log(trx.pub_keys);
keyRes = await eosio.dfuseClient.stateKeyAccounts(publicKey, {block_num: blockNum});
console.log(keyRes); // Error thrown before here, key is not found

See line 93 and line 121 of OrvView.js. This shows the following output where we can see the public keys are not the ones corresponding to the signing key.

transfer Object { from: "gov", memo: "", quantity: "3000.00 EUR", to: "thenewfork" }
EOS7jdsDcTShSkZ6E32WXfrFk4aQ3ARAq1pVP71XFob8rj1dSgtJe
transfer Object { from: "thenewfork", memo: "", quantity: "600.00 EUR", to: "jack" }
EOS4vcU9MFRajXgNsxp9qcnFuCLcjKZ1EXebAgqFKv9pEXicYCdLX
transfer Object { from: "thenewfork", memo: "", quantity: "600.00 EUR", to: "kirsten" }
EOS81jUNCBWyW9gm8oeTArYp5PdJdCvg7vDYRkepJWjfSR5VWzyjt
transfer Object { from: "thenewfork", memo: "", quantity: "600.00 EUR", to: "matej" }
EOS7c7VXAdEAMeNcAqHvx6LtGcnGquYPGFGRPCaqoiTU4QLj7zLMC

We are running dfuse in docker:
https://github.com/Conscious-Cities/eosio-react-app/blob/a738d786d716f4ce9994794aa715b06ea79758fd/blockchain/Dockerfile-dfuse

Merge and store bottleneck on local sync

I performed a sync of Proton chain from block 0 to block 3M, then hit Ctrl-C. It tooks 25m between nodeos termination and when dfuse for EOSIO actually quit. This is because the blocks chan channel was at max capacity of 100K blocks in it. So it took roughly 25m to drain in and create the merged-blocks file from that.

Not sure when I reached that exactly, but it probably hinder my syncing performance with the Proton chain because I imagine was full relatively rapidly. Opening so we gather feedback about:

Possibilities of improvements
Won't fix case due to possibility to perform "parallel" sync

Relevant files:

start:
  args:
  - mindreader
  flags:
    mindreader-log-to-zap: false
    mindreader-merge-and-store-directly: true
    mindreader-start-failure-handler: true
    mindreader-blocks-chan-capacity: 100000
    mindreader-restore-snapshot-name: latest

build instructions for dfuse-eosio don't work on develop branch

Trying to follow the build instructions here. https://github.com/dfuse-io/dfuse-eosio

(log files sent privately)

Object storage should use sub-paths, to avoid 100k+ files in the same directory.

one-block files, as well as merged block files, can be written to local filesystem.

Some filesystems don't like 10k+ files in a single directory.

All components could simply handle '/' in the filenames, like:

s3://path/to-network/00100054300.dbin.zstd -> s3://path/to-network/001/000/54300.dbin.zstd

A dfuseeos tools function could rename those files in one swift, and bstream's FileSource, the merger and mindreader (writing those files), could be adapted to understand (even both methods).

dstore's Walk() method needs to be adapted, to dig into subdirectories without a hiccup. (I do think it's already going to do this flawlessly, on all backends, so perhaps just a quick check).

eosq account page table tab showing results from previous search

After looking up an account's table, the table's data rendered will persist on the next account lookup.

4-6s search transaction query time with http API

We are running dfuse in a docker container:
https://github.com/Conscious-Cities/eosio-react-app/blob/a738d786d716f4ce9994794aa715b06ea79758fd/blockchain/Dockerfile-dfuse

When we request a transaction search using the https://docs.dfuse.io/reference/eosio/rest/search-transactions API the response typically takes 3-6s, higher than expected. This is on a fresh chain that has been running for only a 5-10 minutes.

In the container, nodeos produces blocks and runs state history plugin and state history api plugin
dfuseos runs all services. I'm running this in a Virtualbox Ubuntu 18.04 machine with 9500Mb RAM and 6 processors. We are calling the dfuse API from a Nodejs docker using node-fetch.

What can we do to speed up the request?

Dedupe and clean-up flags

A few issues:

-indices-store are duplicated (between search-archive and search-indexer)
-blocks-store are duplicated (between almost all apps, blockmeta, eosws, fluxdb, kvdb, mindreader, relayer, search-live, search-forkresolver)
--mindreader-oneblock-store-url and --merger-one-block-path are named differently, but refer to the same concept (could be deduped)

If we were to dedupe, then we'd lose the knowledge of which app consumes that flag. How could that be communicated otherwise? Is it important?

I propose a flag called --common-blocks-store instead.

SQL Sync proposal

First draft proposal, please comment on.

Here's what a SQL Sync config could look like:

sqlsync:
  tables_prefix: "eosio_chain1_"  # Global prefix, for all tables

  whitelist_contracts:
    "*":
      tables: "*"

    # - or -

    eosio.token:
      tables: ["table1", "table2", "table3"]

  blacklist_contracts:
  - skipcontract
  - skipcontract2
  - skipcontract3

  mappings:
  - label: "All currency contracts"

    # Select which contracts, tables, these mappings apply to
    contracts: "*"
    scopes: "*"
    tables: ["accounts"]

    # If set to true, use:
    #    [prefix]_[contract]_[table]_[scope]
    # instead of:
    #    [prefix]_[contract]_[tableName]
    scoped_table_name: false  # Mutually opposed to `scope_as_field`.
    scope_as_field: true  # Creates a compound primary key (_key, _scope)

    # If true, include the contract name in the SQL table name
    # Otherwise, strips  `[contract]_` from SQL tables names.
    full_table_name: true  

    # Some mapping definitions
    sql_type_by_field:
      amount: "character varying(26)"
      memo: "varchar(255) | STRING(?)"
    sql_type_by_type:  # For other fields
      string: "varchar(64)"

  - label: "Positions"
    contracts: ["nav_past", "nav_going"]
    scopes: "*"
    tables: ["positions"]

    # Some ways of combining some fields into one when in SQL, etc..
    sql_type_by_field:
      lat: "character varying(26) | POINT({lat}, {lon})"
      lon: "SKIP"
    sql_type_by_type:  # For other fields
      string: "varchar(64)"

dstore: `badger` needs to MkdirAll before calling Open()

Git commit: a1cdafecdcebd8b3148ca290cb26d11b662d81cb

Steps:

$ go install ./cmd/dfuseeos
$ mkdir -p /tmp/test
$ cd /tmp/test
$ dfuseeos init (Answer yes to run producer)
$ dfueeos start

Starting dfuse for EOSIO with config file './dfuse.yaml'
Launching applications: abicodec,apiproxy,blockmeta,dashboard,dgraphql,eosq,eosws,fluxdb,kvdb-loader,merger,mindreader,node-manager,relayer,search-archive,search-indexer,search-live,search-router
unable to launch (cli/start.go:102) {"error": "unable to create app \"blockmeta\": badger new: open badger db: badger new: open badger db: Error Creating Dir: \"/tmp/test/dfuse-data/kvdb/kvdb_badger.db\": mkdir /tmp/test/dfuse-data/kvdb/kvdb_badger.db: no such file or directory"}

Lot's of warning message when starting a fresh local chain now

Two places needed to be adjusted:

nodeos/snapshot.go complains it cannot find snapshot in warning, but it's a fresh chain. Message is in warning and it's misleading because we talk about a replay, which it not the case here. I suggest we move that to INFO and improve the message
Now that nodeos maps it's warning to zap level, it's annoying to have them by default. We should probably try to tweak it to have ERROR only. That might prove harder than it seems because the level is at the app level while we would need to tweak it for a particular package here.

Note also that I was not able to choose a label for this issue, no label seems to fit here, where is the enhancement like label?

$ ./dfuseeos start
Starting dfuse for EOSIO with config file './dfuse.yaml'
Launching applications: abicodec,apiproxy,blockmeta,dashboard,dgraphql,eosq,eosws,fluxdb,merger,mindreader,node-manager,relayer,search-archive,search-indexer,search-live,search-router,trxdb-loader
Your instance should be ready in a few seconds, here some relevant links:

  Dashboard:        http://localhost:8081

  Explorer & APIs:  http://localhost:8080
  GraphiQL:         http://localhost:8080/graphiql

Cannot find latest snapshot, will replay from blocks.log (nodeos/snapshot.go:153)
warn  2020-05-08T21:58:27.837 thread-0  platform_timer_accurac:64     compute_and_print_ti ] Checktime timer accuracy on this platform and hardware combination is poor; accuracy of subjective transaction deadline enforcement will suffer (log_plugin/to_zap_log_plugin.go:53)
warn  2020-05-08T21:58:27.856 thread-0  controller.cpp:583            startup              ] No existing chain state or fork database. Initializing fresh blockchain state and resetting fork database (log_plugin/to_zap_log_plugin.go:53)
warn  2020-05-08T21:58:27.857 thread-0  controller.cpp:452            initialize_blockchai ] Initializing new blockchain with genesis state (log_plugin/to_zap_log_plugin.go:53)
Cannot find latest snapshot, will replay from blocks.log (nodeos/snapshot.go:153)
warn  2020-05-08T21:58:28.699 thread-0  platform_timer_accurac:64     compute_and_print_ti ] Checktime timer accuracy on this platform and hardware combination is poor; accuracy of subjective transaction deadline enforcement will suffer (log_plugin/to_zap_log_plugin.go:53)
warn  2020-05-08T21:58:28.706 thread-0  controller.cpp:583            startup              ] No existing chain state or fork database. Initializing fresh blockchain state and resetting fork database (log_plugin/to_zap_log_plugin.go:53)
warn  2020-05-08T21:58:28.706 thread-0  controller.cpp:452            initialize_blockchai ] Initializing new blockchain with genesis state (log_plugin/to_zap_log_plugin.go:53)

Dashboard - display ports

Would be handy to display the relevant ports for each component within the dashboard. We'll help to make it more of a one-stop tool to gather information.

"dfuseeos --log-format text" should be tty aware

dfuseeos --log-format text should be tty aware.

If you run dfuseeos under systemd, dfuseeos still tries to output color highlighting which doesn't work well in syslog.

Dfuse dockerfile does not work when apt-get clean is run

I have created the following dockerfile to run all dfuse services (thanks to @mmcs85 for original code here https://github.com/operanditech/dfuseup/blob/master/image/Dockerfile)

FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y --no-install-recommends wget ca-certificates software-properties-common
RUN wget https://github.com/EOSIO/eosio.cdt/releases/download/v1.7.0/eosio.cdt_1.7.0-1-ubuntu-18.04_amd64.deb \
    && apt install -y ./eosio.cdt_1.7.0-1-ubuntu-18.04_amd64.deb \
    && rm eosio.cdt_1.7.0-1-ubuntu-18.04_amd64.deb
RUN wget https://github.com/dfuse-io/eos/releases/download/v2.0.3-dm/eosio_2.0.3-dm_ubuntu-18.04_amd64.deb \
    && apt install -y ./eosio_2.0.3-dm_ubuntu-18.04_amd64.deb \
    && rm eosio_2.0.3-dm_ubuntu-18.04_amd64.deb
RUN wget https://github.com/dfuse-io/dfuse-eosio/releases/download/v0.1.0-beta2/dfuse-eosio_0.1.0-beta2_linux_x86_64.tar.gz \
    && tar -xzvf dfuse-eosio_0.1.0-beta2_linux_x86_64.tar.gz \
    && rm dfuse-eosio_0.1.0-beta2_linux_x86_64.tar.gz
RUN mv dfuseeos /usr/local/bin/dfuseeos \
    && apt-get remove -y wget ca-certificates \
    && apt-get autoremove -y \
#   && apt-get clean \
    && rm -rf /var/lib/apt/lists/* \
    && cd /usr/opt/eosio.cdt/1.7.0/bin \
    # Hack to fix multithreading issue in eosio.cdt 1.7.0
    && mv wasm-ld wasm-ld.real \
    && echo '#!/bin/bash' > wasm-ld \
    && echo '`dirname $0`/wasm-ld.real --no-threads "$@"' >> wasm-ld \
    && chmod +x /usr/opt/eosio.cdt/1.7.0/bin/wasm-ld
RUN cd / \
    # "y" will initialize to run a producing node, "n" will connect to existing node
    && (echo "y" && cat) | dfuseeos init
RUN sed -i 's/127.0.0.1:8888/0.0.0.0:8888/g' producer/config.ini

which I then run with # docker run -p 8888:8888 -p 8081:8081 -p 8080:8080 -it {{image_name_or_id}} dfuseeos start

If the line apt-get clean is commented out (as it is above) then all is fine and dfuse starts and services can be accessed on port 8888, 8080 and 8081

If this line is not commented out then the following occurs when run
got err on get into (nodeos/monitoring.go:44) {"error": "http://:8888/v1/chain/get_info: Post "http://:8888/v1/chain/get_info\": dial tcp :8888: connect: connection refused"} Received termination signal, quitting

Data migration tooling

MVP

Limbo (MVP or Nice To Have)

Correctly set table payer quirks (logic to correctly make the good user pay for table creation on contracts)

Nice To have/Second Phase

Performance testing of time to export data
Performance testing & tweaks of time to import data (check with @abourget BIOS parameters, better packed actions into transaction, measurements, etc.)
Documentation about the app and how user can use it
- Document quirks about requirements (eosio key, eosio default contract, etc.)
Exclude/Include accounts/contracts to export/import (to de define the expected behavior)
Able to resume an import where we left off (find way to create "checkpoint", this is harder to implement as it's about reconciling expected state with desired state and execute the difference)

Here's a first draft of the flow:

Export from chain

all accounts (whitelist, blacklist): snapshot at LIB at a given time, through fluxdb:
- the account
  - its permission structs
  - its linked permissions
  - its cpu/net/ram associated resources
- check whether its a contract: for a given block height.
  - whitelist/blacklist of contracts/tables combinations
  - list all tables (caught from the on-chain ABI, fetched since the "all accounts" snapshot)
  - all scopes
  - all rows

Outputs:

large json file for accounts
large json file(s) for tables
dir/{[ac]/[ou]/[acountName]}/tables/[tableName]/{[sc]/[pe]/[scopeName]}.json
example:
dir/dfu/se/dfuse44shine/tables/posts/eo/sc/eoscanadacom.json
dir/{[ac]/[ou]/[acountName]}/codet.wasm
dir/{[ac]/[ou]/[acountName]}/abi.json
dir/{[ac]/[ou]/[acountName]}/account.json
large json of abi + code on current contract accounts

Processing:

make a copy
run some scripts for mutating them:
- could pick up all updated contracts from a given directory (assuming contract/contract.wasm and contract/contract.abi)
  - everything that is left in the output dir, would be injected
  - anything left unchanged, will be passed on to the new chain
- to alter abi, to reflect the migrated ones
- to alter code
- to alter table contents

Import phase:

Mapping between how we want to apply accounts changes.
- Series of transactions: newaccount
  - Set temporary key for updateauth, so we can apply any contract changes.
For table data: code generation
- For all contracts:
  - setabi to the NEW abi (after mutations)
  - setcode to insert_row.wasm
  - send transactions for ALL ROWS in the mutated json files, for the given contract
    - one-by-one, packed with the corresponding ABI
    - signer would need to be the same as the payer for the rows being inserted
      - OPT1: what about TABLE creation and billing? Who was the first creator in the original chain?
        
        refinement:
        
        get the payer for the TABLE
        
        find a row that is from that payer
        
        make it the first write.
      - OPT2: we write the first row, by altering its payer for the TABLE payer
        
        we REWRITE that same first row, tweaking only its payer for the actualy payer of the row
        
        continue on merrily
  - setcode to the NEW code (after mutations)
Go through accounts again:
- Remove the temporary authorization
- Apply the updateauth, linkauth, setalimits (or setacctram, setacctnet, setacctcpu)
update_row from Argentina needs ABI packing
- https://github.com/EOSArgentina/Mallmann/blob/master/code/update_row/update_row.c
eosio.setcode(update_row.wasm)
targetcontract.whatever(binarypackeddatato write to the given row)
eosio.setcode(original.wasm)

Proposal for Partial synchronization

Context

Historically, all dfuse components were created with the assumption that the whole blockchain would be processed. A lot of effort has been put to allow parallel reprocessing of all components (block files, database, etc.)

Now that users are running local instances of dfuse for EOSIO, it makes sense to allow them to sync a chain only partially, from a specific block (usually close to HEAD). Parts of it can already be done, but there are limitations.

What works currently

With this pull request: #47,

You can sync with kylin within 1h (depending on the age of your source state snapshot)
You can have a working 'search' in a given range (graphql) -- without decoded dbops --
You can have a working "live" search on the head (graphql) -- without decoded dbops --
You can get "block" or get "blockref by time range" (graphql)

What does NOT work currently

Fluxdb currently requires syncing from the beginning of the chain, because it saves ALL historical changes. This would mean generating ALL block files (nodeos+mindreader running from block 1, or in parallel from different snapshots...)
eosws (serving all websocket/REST) has not been tested with partial chain...
abicodec requires a working fluxdb, this prevents decoding the dbops on search results

Proposal

FluxDB could bootstrap from a state snapshot (decoding all objects directly from that file). It would require FluxDB to have the concept of a "low_block_num", at which it has that snapshot.
The hardest part is the decoding from the state snapshots, the behavior modifications in fluxdb should be trivial.

This would allow abicodec to work

(Eosws has some calls depending on fluxdb, more discovery work is needed.)

eosq shows `Cannot query field "accountBalances" on type "Query".`

Thanks for your awesome work!

When I search for an account using eosq, for example eosio, the page shows Cannot query field "accountBalances" on type "Query".

dfuseeos help tries to write to logs. don't open log file for handling help

Unable to use "dfuse-data/dfuse.log.json" for logging purposes, trying with "/tmp/dfuse.log.json" instead (error: couldn't open sink "dfuse-data/dfuse.log.json": open dfuse-data/dfuse.log.json: no such file or directory)
Unable to use "/tmp/dfuse.log.json" for logging purposes, logs won't be saved in a log file and will be printed to console only (error: couldn't open sink "dfuse-data/dfuse.log.json": open dfuse-data/dfuse.log.json: no such file or directory)
dfuse for EOSIO

Usage
...

Sync'ing a large chain

See https://docs.dfuse.io/eosio/admin-guide/large-chains-preparation/

PARTIAL_SYNC.md needs updating for EOS Nation snapshots instructions

https://github.com/dfuse-io/dfuse-eosio/blob/develop/PARTIAL_SYNC.md

snapshots are now in zstd format (was previously bzip2)

FluxDB: Should implement the graceful shutdown pattern

FluxDB server at the moment simply waits 5 seconds on shutdown before close the HTTP server. It should implement a GracefulShutdown with the derr library to allow a delay before actually shutting down

--skip-checks not recognized as a valid flag

When running dfuseeos with --skip-checks it will show

Error: unknown flag: --skip-checks
Usage:
  dfuseeos start [flags]
...

~~the checks are skipped though~~

version
dfuseeos version 0.1.0-beta3-a45e062c6482a1c9765d5195cb451aed1a67e662

implement new feature mindreader 'dumb-mode'

This feature would have mindreader do nothing about managing snapshots or playing around with nodeos RPC or anything. It would allow this usage:

nodeos | dfuse mindreader

Anyone using this mode would manage everything in nodeos outside of dfuse.

Support `boot` command, and kickstart a full-blown chain.

I propose we tweak init so it renders:

run:
  start:
  - boot
  - mindreader
  ...
  args:
   - --boot-if-empty=./bootseq.yaml

and the addition of:

dfuseeos boot ./path/to/bootseq.yaml  http://localhost:8888

and perhaps:

dfuseeos boot --preset mainnet --boot-options
dfuseeos boot --preset scratch

which uses eos-bios to fill-in your chain with something meaningful when you start. It would do it optionally, probably by checking if eosio has a contract in place the moment mindreader is up and running. The boot app could stay there and do nothing else after it has done its checks.

See also #54 which would benefit from this.

add s3 bucket support for storing blocks (one-blocks and hundred-blocks)

Allow blocks to be stored in an s3 bucket as an option. eg minio.

If using a simple s3 server, then consider also this issue
#56

update dfuseeos to eosio 2.0.6

block.one added some nice features in 2.0.6 (default thread priority changes). It would be great to have a dfuseeos binary based on that 2.0.6

for dfuseeos -log-format text to output anything you need to set environment

for dfuseeos -log-format text to output anything you need to set environment. However environment is not documented anywhere

export INFO=mindreader

would expose mindreader info logging. I suppose INFO takes a comma separated list of modules?

eosq not accessible across the network

Seems to attach back to localhost all the time. Could use window.location or a relative path?

Reported by EOSUSA Michael on Telegram

Support "mounting" apps' URL paths

Right now, if you have a rewriting proxy in front of eosq or the dashboard (routing /eosq -> to the port of dfuseeos serving eosq), the returned HTML will not consider the /eosq mount point and provide wrong links (to things like /static or /manifest.json), so when the rewriting proxy is hit, it does not know to route /manifest.json to the same backend.

Affects K8s Ingress routing, Apache or nginx routing, etc..

We would need something like a --eosq-mount-point that would be taken in consideration when serving the app... pass that as a config variable when rendering the index.html for the React app. And use something like: https://stackoverflow.com/questions/34164103/server-rendering-react-app-behind-a-url-prefix-with-react-router so that the react-router is aware the app is being served from a prefixed URL.

Mindreader failed to start nodeos process after forced exit

Environment
This runs dfuse using the docker image shown here #20
dfuse v0.1.0-beta2
eosio v2.0.3
eosio-cdt v1.7.0

To reproduce

Run dfuseeos dfuseeos start
Stop the process with Ctr+C
Run dfuseeos dfuseeos start

Expected
Dfuse will start its services correctly after it was shut down

Actual
The following is displayed and the process exits with code 1

got err on get into (nodeos/monitoring.go:44) {"error": "http://:8888/v1/chain/get_info: Post \"http://:8888/v1/chain/get_info\": dial tcp :8888: connect: connection refused"}
notifying starting failure handler because process exited with non-success code within 10 seconds (superviser/superviser.go:167)
*********************************************************************************
* Mindreader failed to start nodeos process
* To see nodeos logs...
* DEBUG=\"github.com/dfuse-io/manageos.*\" dfuseeos start
*********************************************************************************

Make sure you have a dfuse instrumented 'nodeos' binary, follow instructions
at https://github.com/dfuse-io/dfuse-eosio/blob/develop/INSTALL.md
to find how to install it (cli/apps.go:208)

These are the last log lines:

{"level":"info","ts":1589905629.282772,"logger":"merger","caller":"[email protected]/merger.go:311","msg":"retrieved list of files","seenblock_low_boundary":0,"bundle_lower_block":100,"seen_files_count":0,"too_old_files_count":0,"good_files_count":24}
{"level":"info","ts":1589905629.2829127,"logger":"merger","caller":"[email protected]/merger.go:248","msg":"waiting for more files to complete bundle","bundle_lowerblock":100,"bundle_length":24,"bundle_upper_block_id":""}
{"level":"info","ts":1589905629.2831578,"logger":"search-router","caller":"router/router.go:91","msg":"routing active query","trace_id":"bbbf60e391efa894158414a7b3e71559","request":{"query":"receiver:eosio action:setabi notif:false","lowBlockNum":1,"highBlockUnbounded":true,"withReversible":true,"liveMarkerInterval":1},"head_delay_tolerance":0,"lib_delay_tolerance":0}
{"level":"info","ts":1589905629.2832675,"logger":"search-router","caller":"router/router.go:98","msg":"request received while router is not ready"}
{"level":"info","ts":1589905629.283731,"logger":"abicodec","caller":"abicodec/syncer.go:83","msg":"the search stream ended with error","error":"search stream terminated with error: rpc error: code = Unavailable desc = search is currently unavailable, try again shortly."}
{"level":"info","ts":1589905629.2837892,"logger":"abicodec","caller":"abicodec/syncer.go:86","msg":"waiting before startng ABI syncer"}
{"level":"info","ts":1589905629.295431,"logger":"blockmeta","caller":"[email protected]/server.go:114","msg":"cannot get stream's headinfo, retrying","error":"rpc error: code = Unavailable desc = not ready"}
{"level":"info","ts":1589905629.2972982,"logger":"relayer","caller":"[email protected]/relayer.go:137","msg":"cannot get headinfo from backend","address":":13010","error":"rpc error: code = Unavailable desc = not ready"}
{"level":"info","ts":1589905629.2974823,"logger":"relayer","caller":"[email protected]/relayer.go:98","msg":"cannot get head info mindreader, retrying forever","sources_addr":[":13010"]}
{"level":"info","ts":1589905629.3935611,"logger":"mindreader","caller":"superviser/superviser.go:165","msg":"command terminated","status":{"Cmd":"nodeos","PID":37,"Exit":254,"Error":{"Stderr":null},"StartTs":1589905623292244221,"StopTs":1589905629393467579,"Runtime":6.101223514,"Stdout":null,"Stderr":null}}
{"level":"warn","ts":1589905629.393886,"logger":"mindreader","caller":"superviser/superviser.go:167","msg":"notifying starting failure handler because process exited with non-success code within 10 seconds"}
{"level":"error","ts":1589905629.3939543,"logger":"dfuse","caller":"cli/apps.go:208","msg":"*********************************************************************************\n* Mindreader failed to start nodeos process\n* To see nodeos logs...\n* DEBUG=\\\"github.com/dfuse-io/manageos.*\\\" dfuseeos start\n*********************************************************************************\n\nMake sure you have a dfuse instrumented 'nodeos' binary, follow instructions\nat https://github.com/dfuse-io/dfuse-eosio/blob/develop/INSTALL.md\nto find how to install it."}

Separate Producer and Mindreader head block number and drift

Currently the mindreader and the producer, both register a head block number & drift with the name "manager". This has as effect that dfuse-eosio cannot distinguish between both application for the metric value head_block_number & head_block_time_drift. We need to configure the name of the Head block number and drift metric on runtime.

while running dfuse-eosio locally, open your browser @ localhost:9102. Currently you should see

...
head_block_number{app="manager"} 170
...

the goal would be to see

...
head_block_number{app="producer"} 170
head_block_number{app="mindreader"} 172
...

PR: streamingfast/node-manager#1

BUG Failed to get transactions: key not found

Brand new chain, local producer, opening eosq gives blank transactions +

unable to fullfil request (derr/http.go:39) {"trace_id": "52e50819338cf5952a61623bc345a2b8", "error": "failed to get transactions: Key not found", "errorVerbose": "Key not found\ngithub.com/dgraph-io/badger/v2.init\n\t/home/stepd/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/errors.go:36\nruntime.doInit\n\t/usr/lib/go-1.14/src/runtime/proc.go:5414\nruntime.doInit\n\t/usr/lib/go-1.14/src/runtime/proc.go:5409\nruntime.doInit\n\t/usr/lib/go-1.14/src/runtime/proc.go:5409\nruntime.doInit\n\t/usr/lib/go-1.14/src/runtime/proc.go:5409\nruntime.main\n\t/usr/lib/go-1.14/src/runtime/proc.go:190\nruntime.goexit\n\t/usr/lib/go-1.14/src/runtime/asm_amd64.s:1373\nfailed to get transactions\ngithub.com/dfuse-io/derr.Wrap\n\t/home/stepd/go/pkg/mod/github.com/dfuse-io/[email protected]/wrap.go:38\ngithub.com/dfuse-io/dfuse-eosio/eosws/rest.ListTransactionsHandler.func1\n\t/home/stepd/repos/dfuse-eosio/eosws/rest/list_transactions.go:53\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/go-1.14/src/net/http/server.go:2012\ngithub.com/dfuse-io/dmetering.(*MeteringMiddleware).ServeHTTP\n\t/home/stepd/go/pkg/mod/github.com/dfuse-io/[email protected]/middleware.go:52\ngithub.com/dfuse-io/dfuse-eosio/eosws.RESTTrackingMiddleware.func1\n\t/home/stepd/repos/dfuse-eosio/eosws/middleware.go:67\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/go-1.14/src/net/http/server.go:2012\ngithub.com/dfuse-io/dfuse-eosio/eosws.(*AuthFeatureMiddleware).Handler.func1\n\t/home/stepd/repos/dfuse-eosio/eosws/middleware.go:127\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/go-1.14/src/net/http/server.go:2012\ngithub.com/dfuse-io/dauth/authenticator/middleware.(*AuthMiddleware).Handler.func1\n\t/home/stepd/go/pkg/mod/github.com/dfuse-io/[email protected]/authenticator/middleware/middleware.go:61\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/go-1.14/src/net/http/server.go:2012\ngithub.com/dfuse-io/dfuse-eosio/eosws.PreTrackingMiddleware.func1\n\t/home/stepd/repos/dfuse-eosio/eosws/middleware.go:92\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/go-1.14/src/net/http/server.go:2012\ngithub.com/dfuse-io/logging.(*Handler).ServeHTTP\n\t/home/stepd/go/pkg/mod/github.com/dfuse-io/[email protected]/middleware.go:99\ngo.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\t/home/stepd/go/pkg/mod/[email protected]/plugin/ochttp/server.go:92\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/stepd/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212\ngithub.com/gorilla/handlers.(*cors).ServeHTTP\n\t/home/stepd/go/pkg/mod/github.com/gorilla/[email protected]/cors.go:54\ngithub.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/home/stepd/go/pkg/mod/github.com/gorilla/[email protected]/compress.go:148\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/go-1.14/src/net/http/server.go:2012\nnet/http.serverHandler.ServeHTTP\n\t/usr/lib/go-1.14/src/net/http/server.go:2807\nnet/http.(*conn).serve\n\t/usr/lib/go-1.14/src/net/http/server.go:1895\nruntime.goexit\n\t/usr/lib/go-1.14/src/runtime/asm_amd64.s:1373"}

allow search indexer to start without a working blockmeta

if blockmeta is not working (ex: not ready yet, plain not running because we are reprocessing different parts or if the trxdb is not yet filled up), the search indexer refuses to start, because it's trying to get an irreversible ID for "startBlock-1"

Using the new DPoSLIBNumAtBlockHeightFromBlockStore, it could fall back to that behavior.

Adjust some nodeos log levels to be better aligned with ours

(split task from #22 )
Now that nodeos maps it's warning to zap level, it's annoying to have them by default.

Previous suggestion: We should probably try to tweak it to have ERROR only
New suggestion: What if we took the most common "warning" logs and (in a nice hard-coded regex fashion), toned them down to "info" ?

This is open for discussion..

$ ./dfuseeos start
Starting dfuse for EOSIO with config file './dfuse.yaml'
Launching applications: abicodec,apiproxy,blockmeta,dashboard,dgraphql,eosq,eosws,fluxdb,merger,mindreader,node-manager,relayer,search-archive,search-indexer,search-live,search-router,trxdb-loader
Your instance should be ready in a few seconds, here some relevant links:

  Dashboard:        http://localhost:8081

  Explorer & APIs:  http://localhost:8080
  GraphiQL:         http://localhost:8080/graphiql

warn  2020-05-08T21:58:27.856 thread-0  controller.cpp:583            startup              ] No existing chain state or fork database. Initializing fresh blockchain state and resetting fork database (log_plugin/to_zap_log_plugin.go:53)
warn  2020-05-08T21:58:27.857 thread-0  controller.cpp:452            initialize_blockchai ] Initializing new blockchain with genesis state (log_plugin/to_zap_log_plugin.go:53)
Cannot find latest snapshot, will replay from blocks.log (nodeos/snapshot.go:153)
warn  2020-05-08T21:58:28.699 thread-0  platform_timer_accurac:64     compute_and_print_ti ] Checktime timer accuracy on this platform and hardware combination is poor; accuracy of subjective transaction deadline enforcement will suffer (log_plugin/to_zap_log_plugin.go:53)
warn  2020-05-08T21:58:28.706 thread-0  controller.cpp:583            startup              ] No existing chain state or fork database. Initializing fresh blockchain state and resetting fork database (log_plugin/to_zap_log_plugin.go:53)
warn  2020-05-08T21:58:28.706 thread-0  controller.cpp:452            initialize_blockchai ] Initializing new blockchain with genesis state (log_plugin/to_zap_log_plugin.go:53)

FluxDB consumable in streaming + specifying boundaries

There's a few things that would be desired:

GraphQL exposure, in streaming
Implement lower_bound, to continue when things drop

The current JSON is streaming from a server's perspective, but because it arrives as a list, it would require some funky streaming-JSON reader on the client. Otherwise, it needs to be buffered.

FluxDB has no problem streaming out 10 GB, but the client might not be in that situation. A GraphQL streaming snapshot would allow the consumer to handle disconnections, and using the lower_bound, continue where it left off, alongside the capacity to not buffer the whole response.

Check about indexing WAX `simpleassets` contract fields

When we initially settled the indexed fields for the search, we added most eosio.token contract fields.

WAX NFT contract simpleassets is kind of the same now. It's been requested twice already, one time by some of developers around the official WAX and Syed from bloks.io.

The requested fields was assetids which seems to be a good idea to have indexed. We should probably check the contract more and check which other fields make sense to index there.

Bleve AnalyisWorker seems to be started even when no search app is launched

Version: 0.1.1
Commit: 56eb705

While debugging an issue with fluxdb, and rendering the goroutine for a running dfuseeos process that was started with this config:

start:
  args:
  - fluxdb
  flags:
    fluxdb-enable-inject-mode: false
    fluxdb-enable-server-mode: true
    fluxdb-enable-pipeline: false
    fluxdb-http-listen-addr: :9090
    fluxdb-statedb-dsn: bigkv://<dfuse-bt-instance>/eos-kylin-flux-v8?createTable=false

We see those being active as a goroutine:

4 @ 0x4039110 0x4049313 0x4e397a2 0x4069721
#	0x4e397a1	github.com/blevesearch/bleve/index.AnalysisWorker+0x101	/Users/maoueh/work/pkg/mod/github.com/blevesearch/[email protected]/index/analysis.go:102

This is strange because I did not expect any bleve goroutine to exist.

Mindreader `sudog with non-nil elem` with non-nil elem

EOS Mainnet mindreader panicked with fatal error: runtime: sudog with non-nil elem. This comes from dfuse-eosio/codec/consolereader.go:109:

File dfuse-eosio/codec/consolereader.go
106: func (l *ConsoleReader) Read() (out interface{}, err error) {
107: 	ctx := l.ctx
108: 
109: 	for line := range l.readBuffer {
110: 		line = line[6:]
111:

Stacktrace

fatal error: runtime: sudog with non-nil elem

goroutine 165 [running]:
runtime.throw(0x1e53d78, 0x20)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc00fb71c68 sp=0xc00fb71c38 pc=0x440282
runtime.releaseSudog(0xc00e5ba300)
	/usr/local/go/src/runtime/proc.go:360 +0x387 fp=0xc00fb71cd8 sp=0xc00fb71c68 pc=0x4434c7
runtime.chanrecv(0xc000096fc0, 0xc000749e20, 0xc000749e01, 0xc000000101)
	/usr/local/go/src/runtime/chan.go:539 +0x35b fp=0xc00fb71d68 sp=0xc00fb71cd8 pc=0x4115db
runtime.chanrecv2(0xc000096fc0, 0xc000749e20, 0x99)
	/usr/local/go/src/runtime/chan.go:412 +0x2b fp=0xc00fb71d98 sp=0xc00fb71d68 pc=0x41126b
github.com/dfuse-io/dfuse-eosio/codec.(*ConsoleReader).Read(0xc000213f40, 0xc000749e90, 0x40, 0x0, 0xed652f152)
	/work/src/codec/consolereader.go:109 +0x88 fp=0xc00fb71e50 sp=0xc00fb71d98 pc=0xe68a38
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).readOneMessage(0xc0003683f0, 0xc00074c0c0, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:268 +0x38 fp=0xc00fb71eb8 sp=0xc00fb71e50 pc=0x15c3a98
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).ReadFlow(0xc0003683f0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:187 +0xc7 fp=0xc00fb71fd8 sp=0xc00fb71eb8 pc=0x15c27b7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00fb71fe0 sp=0xc00fb71fd8 pc=0x473381
created by github.com/dfuse-io/manageos/mindreader.RunMindReaderPlugin
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:136 +0x69e

goroutine 1 [select, 8293 minutes]:
github.com/dfuse-io/dfuse-eosio/launcher/cli.dfuseStartE(0x46b1440, 0xc0001adb00, 0x1, 0x23, 0x0, 0x0)
	/work/src/launcher/cli/start.go:117 +0xbbc
github.com/spf13/cobra.(*Command).execute(0x46b1440, 0xc0001ad200, 0x23, 0x24, 0x46b1440, 0xc0001ad200)
	/work/go/pkg/mod/github.com/spf13/[email protected]/command.go:838 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0x46b0f00, 0x2d6, 0xc00059fef8, 0x1)
	/work/go/pkg/mod/github.com/spf13/[email protected]/command.go:943 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
	/work/go/pkg/mod/github.com/spf13/[email protected]/command.go:883
github.com/dfuse-io/dfuse-eosio/launcher/cli.Main()
	/work/src/launcher/cli/main.go:61 +0x6b5
main.main()
	/work/src/cmd/dfuseeos/main.go:13 +0x20

goroutine 36 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00012a870)
	/work/go/pkg/mod/[email protected]/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
	/work/go/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57

goroutine 12 [select, 8293 minutes]:
github.com/desertbit/timer.timerRoutine()
	/work/go/pkg/mod/github.com/desertbit/[email protected]/timers.go:119 +0x304
created by github.com/desertbit/timer.init.0
	/work/go/pkg/mod/github.com/desertbit/[email protected]/timers.go:15 +0x35

goroutine 13 [select, 8293 minutes]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc0002af620, 0xc0002af680)
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:94 +0xc8

goroutine 14 [select, 8293 minutes]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc0002af620, 0xc0002af680)
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:94 +0xc8

goroutine 15 [select, 8293 minutes]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc0002af620, 0xc0002af680)
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:94 +0xc8

goroutine 16 [select, 8293 minutes]:
github.com/blevesearch/bleve/index.AnalysisWorker(0xc0002af620, 0xc0002af680)
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:102 +0x102
created by github.com/blevesearch/bleve/index.NewAnalysisQueue
	/work/go/pkg/mod/github.com/fproulx-eoscanada/[email protected]/index/analysis.go:94 +0xc8

goroutine 439 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0004aa190, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000532180, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0006d5200)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:293 +0x10c1

goroutine 64 [select, 1 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000213640)
	/work/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:69 +0xc2
created by google.golang.org/grpc.newCCBalancerWrapper
	/work/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:60 +0x16d

goroutine 95 [select]:
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).consumeReadFlow(0xc0003683f0, 0xc00074c0c0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:224 +0x10c
created by github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).ReadFlow
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:182 +0x87

goroutine 65 [chan receive, 1 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0001efb80)
	/work/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1164 +0x6ea
created by google.golang.org/grpc.(*addrConn).connect
	/work/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:800 +0x128

goroutine 98 [select, 31 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0002138c0)
	/work/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:69 +0xc2
created by google.golang.org/grpc.newCCBalancerWrapper
	/work/go/pkg/mod/google.golang.org/[email protected]/balancer_conn_wrappers.go:60 +0x16d

goroutine 99 [chan receive, 31 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0004a5080)
	/work/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:1164 +0x6ea
created by google.golang.org/grpc.(*addrConn).connect
	/work/go/pkg/mod/google.golang.org/[email protected]/clientconn.go:800 +0x128

goroutine 100 [IO wait, 8292 minutes]:
internal/poll.runtime_pollWait(0x7f39ff554fa0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000696498, 0x72, 0x0, 0x0, 0x1e1e9a3)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000696480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000696480, 0xafbf2ce2899afd01, 0x1000000000000, 0xafbf2ce2899afd01)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000190560, 0x5eb962c2, 0xc000033dd8, 0x4d7a76)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000190560, 0xc000033e28, 0x18, 0xc000103e00, 0x782b3c)
	/usr/local/go/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc0000b6000, 0x370ea40, 0xc000190560, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc0000b6000, 0xc0004a7fb0, 0x0)
	/usr/local/go/src/net/http/server.go:2830 +0xb7
github.com/dfuse-io/dmetrics.Serve(0x1e1b6ab, 0x5)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/http.go:25 +0xab
created by github.com/dfuse-io/dfuse-eosio/launcher/cli.setup
	/work/src/launcher/cli/setup.go:36 +0x66

goroutine 101 [IO wait, 8293 minutes]:
internal/poll.runtime_pollWait(0x7f39ff554ec0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00011eb18, 0x72, 0x0, 0x0, 0x1e1e9a3)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00011eb00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc00011eb00, 0xc00007bc78, 0xc000437880, 0x7f39ff6cce98)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000384440, 0xc00007bcb8, 0x418738, 0x30)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000384440, 0x1c8e7c0, 0xc000798840, 0x1ad3620, 0x469d990)
	/usr/local/go/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc0004d8000, 0x370ea40, 0xc000384440, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc0004d8000, 0xc0004d8000, 0x0)
	/usr/local/go/src/net/http/server.go:2830 +0xb7
net/http.ListenAndServe(...)
	/usr/local/go/src/net/http/server.go:3086
github.com/dfuse-io/dfuse-eosio/launcher/cli.setup.func1()
	/work/src/launcher/cli/setup.go:45 +0x69
created by github.com/dfuse-io/dfuse-eosio/launcher/cli.setup
	/work/src/launcher/cli/setup.go:43 +0x211

goroutine 20883 [runnable]:
internal/poll.runtime_pollWait(0x7f39ff555400, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc005394318, 0x72, 0x8001, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005394300, 0xc005980000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
	/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc011ad5778, 0xc005980000, 0x8000, 0x8000, 0xe8, 0x0, 0x0)
	/usr/local/go/src/os/file.go:116 +0x71
io.copyBuffer(0x36d2140, 0xc0056ec390, 0x36d66e0, 0xc011ad5778, 0xc005980000, 0x8000, 0x8000, 0xc0180f27a0, 0x41115f, 0xc005248cb8)
	/usr/local/go/src/io/io.go:405 +0x122
io.Copy(...)
	/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0xc0180f27b8, 0x7d87e7)
	/usr/local/go/src/os/exec/exec.go:311 +0x63
os/exec.(*Cmd).Start.func1(0xc005367340, 0xc00d968be0)
	/usr/local/go/src/os/exec/exec.go:441 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:440 +0x632

goroutine 440 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0006d5200)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:968 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:302 +0x10e6

goroutine 441 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff5548a0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000696218, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000696200, 0xc00060c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000696200, 0xc00060c000, 0x8000, 0x8000, 0x203003, 0x0, 0x60100000008)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000118018, 0xc00060c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc0002ce300, 0xc00067e3b8, 0x9, 0x9, 0xc00a659880, 0x1007f39ff6cc108, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc0002ce300, 0xc00067e3b8, 0x9, 0x9, 0x9, 0x5ec0fa52, 0xc00532adc8, 0x4d7a76)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc00067e3b8, 0x9, 0x9, 0x36d1280, 0xc0002ce300, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00067e380, 0xc0091cfb60, 0x47782a0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0006d5200, 0xc0004a01b0, 0x3392dc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:458 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc00020e000, 0x372cce0, 0xc0006d5200)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:718 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00020e000, 0x372cce0, 0xc0006d5200)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:678 +0x562

goroutine 428 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff554de0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000660a98, 0x72, 0x0, 0x0, 0x1e1e9a3)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000660a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000660a80, 0x71276b40264ab14d, 0x0, 0x71276b40264ab14d)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0005e5be0, 0x5ec0fa50, 0xc000079de0, 0x4d7a76)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0005e5be0, 0xc000079e30, 0x18, 0xc000022300, 0x782b3c)
	/usr/local/go/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc0000b70a0, 0x370ea40, 0xc0005e5be0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc0000b70a0, 0x0, 0x473381)
	/usr/local/go/src/net/http/server.go:2830 +0xb7
github.com/dfuse-io/manageos/operator.(*Operator).RunHTTPServer.func2(0xc0000b70a0, 0xc0000969c0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/operator/http_server.go:81 +0x2f
created by github.com/dfuse-io/manageos/operator.(*Operator).RunHTTPServer
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/operator/http_server.go:80 +0x1213

goroutine 103 [select, 8293 minutes]:
github.com/dfuse-io/dfuse-eosio/launcher.(*Launcher).Launch.func3(0xc000200880, 0x1e23b19, 0xa, 0x371ba40, 0xc0002c7020)
	/work/src/launcher/launcher.go:123 +0xd3
created by github.com/dfuse-io/dfuse-eosio/launcher.(*Launcher).Launch
	/work/src/launcher/launcher.go:122 +0x5b5

goroutine 104 [sleep]:
time.Sleep(0x12a05f200)
	/usr/local/go/src/runtime/time.go:188 +0xba
github.com/dfuse-io/dfuse-eosio/launcher.(*Launcher).Launch.func4(0xc000200880)
	/work/src/launcher/launcher.go:146 +0x45
created by github.com/dfuse-io/dfuse-eosio/launcher.(*Launcher).Launch
	/work/src/launcher/launcher.go:142 +0x647

goroutine 105 [sleep]:
time.Sleep(0x12a05f200)
	/usr/local/go/src/runtime/time.go:188 +0xba
github.com/dfuse-io/dfuse-eosio/metrics.(*Manager).Launch(0xc0007aaa80, 0x1)
	/work/src/metrics/manager.go:146 +0x4f2
created by github.com/dfuse-io/dfuse-eosio/launcher/cli.dfuseStartE
	/work/src/launcher/cli/start.go:112 +0xb23

goroutine 106 [syscall, 8293 minutes]:
os/signal.signal_recv(0x1)
	/usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.Notify.func1
	/usr/local/go/src/os/signal/signal.go:127 +0x44

goroutine 108 [chan receive, 8293 minutes]:
github.com/dfuse-io/derr.SetupSignalHandler.func1(0xc0000a2d20, 0xc0004f8810, 0x0, 0xc0007aaba0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/signals.go:48 +0x55
created by github.com/dfuse-io/derr.SetupSignalHandler
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/signals.go:46 +0x112

goroutine 111 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff555080, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000200b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000200b00, 0xc000095000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000200b00, 0xc000095000, 0x1000, 0x1000, 0x4, 0xc00a61f4c0, 0x4)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a420, 0xc000095000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
net/http.(*connReader).Read(0xc0005fb2c0, 0xc000095000, 0x1000, 0x1000, 0x4f1944, 0xc015da17d0, 0x203003)
	/usr/local/go/src/net/http/server.go:786 +0xf4
bufio.(*Reader).fill(0xc0007aad80)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).ReadSlice(0xc0007aad80, 0xa, 0x28, 0xc0002bd9c0, 0x417ec6, 0xc00d279b00, 0x100)
	/usr/local/go/src/bufio/bufio.go:359 +0x3d
bufio.(*Reader).ReadLine(0xc0007aad80, 0xc0002bd9c8, 0xc00a659880, 0x7f39ff6cce98, 0x0, 0x418738, 0x30)
	/usr/local/go/src/bufio/bufio.go:388 +0x34
net/textproto.(*Reader).readLineSlice(0xc00d0b5f50, 0xc00d279b00, 0x0, 0x442d9c, 0xc0002bda28, 0x0)
	/usr/local/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc0007aad80, 0x0, 0xc00d279b00, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:1015 +0xa4
net/http.(*conn).readRequest(0xc00062a500, 0x3716080, 0xc000213c00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x191
net/http.(*conn).serve(0xc00062a500, 0x3716080, 0xc000213c00)
	/usr/local/go/src/net/http/server.go:1822 +0x6d4
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2933 +0x35c

goroutine 109 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff555160, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00011f098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00011f080, 0xc00002b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00011f080, 0xc00002b000, 0x1000, 0x1000, 0x442d9c, 0xc000657b70, 0x46fd30)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a418, 0xc00002b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
net/http.(*persistConn).Read(0xc0004e6240, 0xc00002b000, 0x1000, 0x1000, 0xc0000a2f60, 0xc000657c70, 0x410715)
	/usr/local/go/src/net/http/transport.go:1825 +0x75
bufio.(*Reader).fill(0xc0007aac60)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc0007aac60, 0x1, 0x0, 0x0, 0x1, 0xc00d164000, 0x0)
	/usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0004e6240)
	/usr/local/go/src/net/http/transport.go:1978 +0x1a8
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1647 +0xc56

goroutine 110 [select]:
net/http.(*persistConn).writeLoop(0xc0004e6240)
	/usr/local/go/src/net/http/transport.go:2277 +0x11c
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1648 +0xc7b

goroutine 164 [runnable]:
bufio.(*Scanner).Scan(0xc00011fb80, 0xc00047ffb8)
	/usr/local/go/src/bufio/scan.go:134 +0x9c3
github.com/dfuse-io/dfuse-eosio/codec.(*ConsoleReader).setupScanner.func1(0xc000213f40)
	/work/src/codec/consolereader.go:68 +0x33
created by github.com/dfuse-io/dfuse-eosio/codec.(*ConsoleReader).setupScanner
	/work/src/codec/consolereader.go:67 +0x141

goroutine 427 [select, 8284 minutes]:
github.com/dfuse-io/manageos/operator.(*Operator).Launch(0xc0000969c0, 0x1, 0x7fff97122992, 0x5, 0xc000010620, 0x1, 0x1, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/operator/operator.go:139 +0x405
created by github.com/dfuse-io/manageos/app/nodeos_mindreader.(*App).Run
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/app/nodeos_mindreader/app.go:223 +0x1183

goroutine 96 [semacquire]:
sync.runtime_Semacquire(0xc00029ea90)
	/usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc00029ea88)
	/usr/local/go/src/sync/waitgroup.go:130 +0x64
golang.org/x/sync/errgroup.(*Group).Wait(0xc00029ea80, 0xc00f65cf80, 0x4)
	/work/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:40 +0x31
github.com/dfuse-io/manageos/mindreader.(*DefaultArchiver).uploadFiles(0xc000213e80, 0xc003e58e00, 0x2)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/archiver.go:149 +0x344
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).alwaysUploadFiles(0xc0003683f0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:208 +0x6b
created by github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).ReadFlow
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:183 +0xac

goroutine 453 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0004be000)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:968 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:302 +0x10e6

goroutine 166 [IO wait, 8293 minutes]:
internal/poll.runtime_pollWait(0x7f39ff555320, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00011fd98, 0x72, 0x0, 0x0, 0x1e1e9a3)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00011fd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc00011fd80, 0x37e11d600, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000111c20, 0xc000032e48, 0xc000032e50, 0x18)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000111c20, 0x3392d88, 0xc00020e000, 0x372aa00, 0xc00019a258)
	/usr/local/go/src/net/tcpsock.go:261 +0x64
google.golang.org/grpc.(*Server).Serve(0xc00020e000, 0x370ea40, 0xc000111c20, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:597 +0x210
github.com/dfuse-io/manageos/mindreader.RunGRPCServer.func1(0xc00020e000, 0x370ea40, 0xc000111c20, 0xc000097020)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/publisher.go:37 +0x43
created by github.com/dfuse-io/manageos/mindreader.RunGRPCServer
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/publisher.go:36 +0x2e8

goroutine 452 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00012a0a0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0002cf380, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0004be000)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:293 +0x10c1

goroutine 7345 [select]:
github.com/dfuse-io/dmetrics.(*HeadTimeDrift).SetBlockTime.func1(0xc000110660)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/head_time_drift.go:60 +0x1da
created by github.com/dfuse-io/dmetrics.(*HeadTimeDrift).SetBlockTime
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/head_time_drift.go:57 +0x8c

goroutine 454 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff5547c0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00011e118, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00011e100, 0xc00079a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00011e100, 0xc00079a000, 0x8000, 0x8000, 0x800010601, 0x0, 0x1060100000000)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000100e8, 0xc00079a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc000128360, 0xc0000b68f8, 0x9, 0x9, 0xc00ab21180, 0x7f39ff6cc7d0, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc000128360, 0xc0000b68f8, 0x9, 0x9, 0x9, 0x9436d5, 0xc00515d72c, 0xc01191e1c0)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0000b68f8, 0x9, 0x9, 0x36d1280, 0xc000128360, 0x0, 0x0, 0xc00515d720, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000b68c0, 0xc00515d720, 0x47782a0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0004be000, 0xc0005dddd0, 0x3392dc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:458 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc00020e000, 0x372cce0, 0xc0004be000)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:718 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00020e000, 0x372cce0, 0xc0004be000)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:678 +0x562

goroutine 20854 [select]:
io.(*pipe).Write(0xc00074c060, 0xc016f9c500, 0xa1, 0x140, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/pipe.go:94 +0x1db
io.(*PipeWriter).Write(...)
	/usr/local/go/src/io/pipe.go:163
github.com/dfuse-io/manageos/mindreader.(*MindReaderPlugin).LogLine(0xc0003683f0, 0xc016a60a00, 0xa0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/mindreader.go:300 +0xaa
github.com/dfuse-io/manageos/superviser.(*Superviser).processLogLine(0xc00066a120, 0xc016a60a00, 0xa0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:185 +0xb7
github.com/dfuse-io/manageos/superviser.(*Superviser).start(0xc00066a120, 0xc005398c60)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:173 +0x1e4
created by github.com/dfuse-io/manageos/superviser.(*Superviser).Start
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:111 +0x48e

goroutine 474 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00060a410, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005f2480, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc003ef8600)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:293 +0x10c1

goroutine 426 [select]:
github.com/dfuse-io/manageos.(*MetricsAndReadinessManager).Launch(0xc000552210)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/monitor.go:54 +0xdc
created by github.com/dfuse-io/manageos/app/nodeos_mindreader.(*App).Run
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/app/nodeos_mindreader/app.go:222 +0x1112

goroutine 433 [select]:
github.com/dfuse-io/bstream/blockstream.(*Server).Blocks(0xc0006145a0, 0xc0000a0870, 0x3726200, 0xc0057ffed0, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/blockstream/server.go:95 +0x420
github.com/dfuse-io/pbgo/dfuse/bstream/v1._BlockStream_Blocks_Handler(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0xc00068bcb0, 0x6)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dfuse/bstream/v1/bstream.pb.go:462 +0x109
github.com/dfuse-io/dgrpc.setupLoggingInterceptors.func2(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0xc00045d440, 0x33904e0, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/logging.go:68 +0xd1
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0x3716140, 0xc00068bbc0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/dfuse-io/dgrpc.setupTracingInterceptors.func2(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0xc00045d440, 0xc00045d460, 0x3716101, 0xc00045d5e0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/tracing.go:39 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc00045d5e0, 0x24, 0xbfa67689b876a45e)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723440, 0xc00045d580, 0xc00045d440, 0xc00045d480, 0xc000318901, 0xc00045d580)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/logging/zap/server_interceptors.go:59 +0x16a
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723440, 0xc00045d580, 0x24, 0xc0000a0820)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723320, 0xc000318940, 0xc00045d440, 0xc00045d4a0, 0xc000010601, 0xc000318940)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 +0xeb
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723320, 0xc000318940, 0x7f39ff6cd560, 0x0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware/tags.StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc00051e180, 0xc00045d440, 0xc00045d4c0, 0xc00045d520, 0xc00045d440)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tags/interceptors.go:38 +0x265
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc00051e180, 0xc00007ac50, 0x418738)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc00051e180, 0xc00045d440, 0x33904e0, 0x3716140, 0xc00068b8c0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58 +0xcf
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00020e000, 0x372cce0, 0xc000023200, 0xc0006d3300, 0xc000552b10, 0x46a2440, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:1244 +0x505
google.golang.org/grpc.(*Server).handleStream(0xc00020e000, 0x372cce0, 0xc000023200, 0xc0006d3300, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:1317 +0xcd6
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000298634, 0xc00020e000, 0x372cce0, 0xc000023200, 0xc0006d3300)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:722 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:720 +0xa1

goroutine 5479217229 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff554980, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc004350f18, 0x72, 0x1000, 0x1001, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc004350f00, 0xc009a16000, 0x1001, 0x1001, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc004350f00, 0xc009a16000, 0x1001, 0x1001, 0x203004, 0x6b7480, 0xc00d295638)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0116b0e58, 0xc009a16000, 0x1001, 0x1001, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc010341ee0, 0xc009a16000, 0x1001, 0x1001, 0x1e, 0xffc, 0xc00532e9c8)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc00d295758, 0x36d1540, 0xc010341ee0, 0x415e15, 0x1b3b2a0, 0x1d9fb40)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc00d295500, 0x36d62a0, 0xc0116b0e58, 0x5, 0xc0116b0e58, 0xd)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00d295500, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc00d295500, 0xc010061000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc00e465ec0, 0xc0054b38b8, 0x9, 0x9, 0xc010061009, 0x4, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc00e465ec0, 0xc0054b38b8, 0x9, 0x9, 0x9, 0xc0001a8535, 0x57b9d2d763d01, 0xc300000000000000)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc0054b38b8, 0x9, 0x9, 0x36d1280, 0xc00e465ec0, 0x0, 0xc000000000, 0x48c1dc, 0xc00522ac10)
	/usr/local/go/src/net/http/h2_bundle.go:1479 +0x87
net/http.(*http2Framer).ReadFrame(0xc0054b3880, 0xc00046f560, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1737 +0xa1
net/http.(*http2clientConnReadLoop).run(0xc00532efa8, 0x4, 0xc00da2cb60)
	/usr/local/go/src/net/http/h2_bundle.go:8246 +0x8d
net/http.(*http2ClientConn).readLoop(0xc00d18b500)
	/usr/local/go/src/net/http/h2_bundle.go:8174 +0x6f
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:7174 +0x64a

goroutine 430 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00060bf90, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005f3aa0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000023200)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:293 +0x10c1

goroutine 431 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000023200)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:968 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:302 +0x10e6

goroutine 432 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff554c20, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0056a0618, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0056a0600, 0xc0006e6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0056a0600, 0xc0006e6000, 0x8000, 0x8000, 0x203004, 0x0, 0x60100000008)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000106d0, 0xc0006e6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc0007ab200, 0xc0000b7458, 0x9, 0x9, 0x4779a60, 0x1007f39ff6cce98, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc0007ab200, 0xc0000b7458, 0x9, 0x9, 0x9, 0x5ec0fa52, 0xc00532ddc8, 0x4d7a76)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0000b7458, 0x9, 0x9, 0x36d1280, 0xc0007ab200, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000b7420, 0xc0090fa8e0, 0x47782a0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000023200, 0xc00068b500, 0x3392dc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:458 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc00020e000, 0x372cce0, 0xc000023200)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:718 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00020e000, 0x372cce0, 0xc000023200)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:678 +0x562

goroutine 466 [select, 8292 minutes]:
github.com/dfuse-io/manageos/superviser.(*Superviser).start(0xc00066a120, 0xc000598580)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:163 +0x16f
created by github.com/dfuse-io/manageos/superviser.(*Superviser).Start
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/superviser/superviser.go:111 +0x48e

goroutine 470 [select]:
github.com/dfuse-io/bstream/blockstream.(*Server).Blocks(0xc0006145a0, 0xc00060a1e0, 0x3726200, 0xc0057cbe40, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/blockstream/server.go:95 +0x420
github.com/dfuse-io/pbgo/dfuse/bstream/v1._BlockStream_Blocks_Handler(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0xc00068abd0, 0x6)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dfuse/bstream/v1/bstream.pb.go:462 +0x109
github.com/dfuse-io/dgrpc.setupLoggingInterceptors.func2(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0xc003e961c0, 0x33904e0, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/logging.go:68 +0xd1
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0x3716140, 0xc00068aae0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/dfuse-io/dgrpc.setupTracingInterceptors.func2(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0xc003e961c0, 0xc003e961e0, 0x3716101, 0xc003e962e0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/tracing.go:39 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x37232c0, 0xc003e962e0, 0x24, 0xbfa67689ea54a395)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723440, 0xc003e96280, 0xc003e961c0, 0xc003e96200, 0xc000318901, 0xc003e96280)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/logging/zap/server_interceptors.go:59 +0x16a
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723440, 0xc003e96280, 0x24, 0xc00060a190)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723320, 0xc000148340, 0xc003e961c0, 0xc003e96220, 0xc00019a001, 0xc000148340)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 +0xeb
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723320, 0xc000148340, 0x7f39ff6cd560, 0x0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware/tags.StreamServerInterceptor.func1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc0000120c0, 0xc003e961c0, 0xc003e96240, 0xc003e96260, 0xc003e961c0)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/tags/interceptors.go:38 +0x265
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc0000120c0, 0xc00065bc50, 0x418738)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x5f
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x1c87ec0, 0xc0006145a0, 0x3723860, 0xc0000120c0, 0xc003e961c0, 0x33904e0, 0x3716140, 0xc00068a900)
	/work/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58 +0xcf
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00020e000, 0x372cce0, 0xc0006d5200, 0xc000662000, 0xc000552b10, 0x46a2440, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:1244 +0x505
google.golang.org/grpc.(*Server).handleStream(0xc00020e000, 0x372cce0, 0xc0006d5200, 0xc000662000, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:1317 +0xcd6
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003b4080, 0xc00020e000, 0x372cce0, 0xc0006d5200, 0xc000662000)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:722 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:720 +0xa1

goroutine 5512207167 [chan receive]:
github.com/dfuse-io/dfuse-eosio/codec.(*decodingQueue).drainQueueFully(0xc0114c4740, 0x738464a)
	/work/src/codec/abi_decoder.go:450 +0x1a8
created by github.com/dfuse-io/dfuse-eosio/codec.newDecodingQueue
	/work/src/codec/abi_decoder.go:378 +0x153

goroutine 475 [select]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003ef8600)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:968 +0x285
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:302 +0x10e6

goroutine 476 [IO wait]:
internal/poll.runtime_pollWait(0x7f39d51ed2a0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000660898, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000660880, 0xc0007a2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000660880, 0xc0007a2000, 0x8000, 0x8000, 0x800010601, 0x0, 0x1060100000000)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a258, 0xc0007a2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
bufio.(*Reader).Read(0xc000532ae0, 0xc000638038, 0x9, 0x9, 0xc00d294a80, 0x7f39ff6cc7d0, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc000532ae0, 0xc000638038, 0x9, 0x9, 0x9, 0x9436d5, 0xc0049a938c, 0xc01192e040)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc000638038, 0x9, 0x9, 0x36d1280, 0xc000532ae0, 0x0, 0x0, 0xc0049a9380, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000638000, 0xc0049a9380, 0x47782a0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003ef8600, 0xc00068b0e0, 0x3392dc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:458 +0x98
google.golang.org/grpc.(*Server).serveStreams(0xc00020e000, 0x372cce0, 0xc003ef8600)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:718 +0xdb
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00020e000, 0x372cce0, 0xc003ef8600)
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
	/work/go/pkg/mod/google.golang.org/[email protected]/server.go:678 +0x562

goroutine 588 [IO wait]:
internal/poll.runtime_pollWait(0x7f39d51ed1c0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000201e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000201e00, 0xc00065c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000201e00, 0xc00065c000, 0x1000, 0x1000, 0x4, 0xc003b98980, 0x4)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0006685f0, 0xc00065c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
net/http.(*connReader).Read(0xc003dd2150, 0xc00065c000, 0x1000, 0x1000, 0x7f39d5bc2450, 0x0, 0x203001)
	/usr/local/go/src/net/http/server.go:786 +0xf4
bufio.(*Reader).fill(0xc00074c000)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).ReadSlice(0xc00074c000, 0xa, 0x7f39d5bc2450, 0xc00068f9c0, 0x417ec6, 0xc00d194400, 0x100)
	/usr/local/go/src/bufio/bufio.go:359 +0x3d
bufio.(*Reader).ReadLine(0xc00074c000, 0xc00068f9c8, 0xc003dc4a80, 0x7f39ff6cd560, 0x0, 0x418738, 0x30)
	/usr/local/go/src/bufio/bufio.go:388 +0x34
net/textproto.(*Reader).readLineSlice(0xc004b5a450, 0xc00d194400, 0x0, 0x442d9c, 0xc00068fa28, 0x0)
	/usr/local/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc00074c000, 0x0, 0xc00d194400, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:1015 +0xa4
net/http.(*conn).readRequest(0xc000081360, 0x3716080, 0xc000555200, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x191
net/http.(*conn).serve(0xc000081360, 0x3716080, 0xc000555200)
	/usr/local/go/src/net/http/server.go:1822 +0x6d4
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2933 +0x35c

goroutine 5512212130 [chan receive]:
cloud.google.com/go/storage.(*Writer).Close(0xc00ce54900, 0x0, 0x0)
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:222 +0x92
github.com/dfuse-io/dstore.(*GSStore).WriteObject(0xc000111780, 0x3716100, 0xc00f3354a0, 0xc00ea41460, 0x2e, 0x36d66e0, 0xc00f6a4160, 0x518872, 0xc00f3354b0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/gsstore.go:68 +0x373
github.com/dfuse-io/dstore.pushLocalFile(0x3716100, 0xc00f3354a0, 0x372f8e0, 0xc000111780, 0xc00ea41440, 0x52, 0xc00ea41460, 0x2e, 0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/common.go:43 +0x23a
github.com/dfuse-io/dstore.(*GSStore).PushLocalFile(0xc000111780, 0x3716100, 0xc00f3354a0, 0xc00ea41440, 0x52, 0xc00ea41460, 0x2e, 0x1, 0xc0000b9518)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/gsstore.go:117 +0x80
github.com/dfuse-io/manageos/mindreader.(*DefaultArchiver).uploadFiles.func1(0x0, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/mindreader/archiver.go:142 +0x259
github.com/abourget/llerrgroup.(*Group).Go.func1(0x100000000000000, 0xc00cde97a0)
	/work/go/pkg/mod/github.com/abourget/[email protected]/group.go:74 +0x33
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00029ea80, 0xc00d1feb40)
	/work/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
	/work/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

goroutine 9502 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff554a60, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000696698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000696680, 0xc0005c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000696680, 0xc0005c8000, 0x1000, 0x1000, 0x4, 0xc00a61f4c0, 0x4)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00019a290, 0xc0005c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
net/http.(*connReader).Read(0xc00f57af60, 0xc0005c8000, 0x1000, 0x1000, 0x7f39b4d99878, 0x0, 0x203002)
	/usr/local/go/src/net/http/server.go:786 +0xf4
bufio.(*Reader).fill(0xc000532e40)
	/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).ReadSlice(0xc000532e40, 0xa, 0x7f39b4d99878, 0xc0002c19c0, 0x417ec6, 0xc00c922a00, 0x100)
	/usr/local/go/src/bufio/bufio.go:359 +0x3d
bufio.(*Reader).ReadLine(0xc000532e40, 0xc0002c19c8, 0xc005256a80, 0x7f39ff6cce98, 0x0, 0x418738, 0x30)
	/usr/local/go/src/bufio/bufio.go:388 +0x34
net/textproto.(*Reader).readLineSlice(0xc00a2f2090, 0xc00c922a00, 0x0, 0x442d9c, 0xc0002c1a28, 0x0)
	/usr/local/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc000532e40, 0x0, 0xc00c922a00, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:1015 +0xa4
net/http.(*conn).readRequest(0xc0007eaaa0, 0x3716080, 0xc011d76400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:966 +0x191
net/http.(*conn).serve(0xc0007eaaa0, 0x3716080, 0xc011d76400)
	/usr/local/go/src/net/http/server.go:1822 +0x6d4
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2933 +0x35c

goroutine 5491356689 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00fff1360, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00ffe2fc0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00acb2fc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:346 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:344 +0xedb

goroutine 5511303334 [select, 1 minutes]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00f6de3c0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00c731c20, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:513 +0x1cd
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00078d6c0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:346 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:344 +0xedb

goroutine 20884 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff555240, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0053943d8, 0x72, 0x8001, 0x8000, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0053943c0, 0xc005988000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
os.(*File).read(...)
	/usr/local/go/src/os/file_unix.go:263
os.(*File).Read(0xc011ad5790, 0xc005988000, 0x8000, 0x8000, 0xf3, 0x0, 0x0)
	/usr/local/go/src/os/file.go:116 +0x71
io.copyBuffer(0x36d2140, 0xc0056ec3c0, 0x36d66e0, 0xc011ad5790, 0xc005988000, 0x8000, 0x8000, 0x370e901, 0xc000482fc8, 0x778293)
	/usr/local/go/src/io/io.go:405 +0x122
io.Copy(...)
	/usr/local/go/src/io/io.go:364
os/exec.(*Cmd).writerDescriptor.func1(0x3393a20, 0xc005396140)
	/usr/local/go/src/os/exec/exec.go:311 +0x63
os/exec.(*Cmd).Start.func1(0xc005367340, 0xc00d968c20)
	/usr/local/go/src/os/exec/exec.go:441 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:440 +0x632

goroutine 5512218863 [runnable]:
github.com/eoscanada/eos-go.(*Decoder).ReadName(0xc0002bf260, 0xc016a41ec0, 0xc, 0x0, 0x0)
	/work/go/pkg/mod/github.com/eoscanada/[email protected]/decoder.go:1099 +0x21c
github.com/dfuse-io/dfuse-eosio/codec.decodeTransfer(0xc016cf9108, 0x21, 0x50ef8, 0x8, 0x0, 0xc00a333ea0, 0x0)
	/work/src/codec/abi_decoder.go:573 +0x17e
github.com/dfuse-io/dfuse-eosio/codec.(*decodingQueue).decodeAction(0xc0114c4740, 0xc016e2a800, 0x1ce94e43e2, 0xc0169ca4c0, 0x40, 0x738464a, 0x0, 0xc00d059980, 0x7f39d5bbd8a0)
	/work/src/codec/abi_decoder.go:482 +0x3b7
github.com/dfuse-io/dfuse-eosio/codec.(*decodingQueue).executeDecodingJob(0xc0114c4740, 0x37160c0, 0xc000198020, 0xc016a56150, 0x1, 0x1, 0xc00b94c7a0, 0x91ab24, 0xc011274000, 0xc004f2a000, ...)
	/work/src/codec/abi_decoder.go:434 +0x523
github.com/dfuse-io/dhammer.(*Hammer).processBatch(0xc009aaa500, 0x37160c0, 0xc000198020, 0xc016a56150, 0x1, 0x1, 0xc016a4d0e0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:164 +0xae
created by github.com/dfuse-io/dhammer.(*Hammer).runInput
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:152 +0x41a

goroutine 5512207165 [select]:
github.com/dfuse-io/dhammer.(*Hammer).runInput(0xc009aaa500, 0x37160c0, 0xc000198020)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:143 +0x38b
created by github.com/dfuse-io/dhammer.(*Hammer).Start
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:93 +0x57

goroutine 5491356688 [IO wait]:
internal/poll.runtime_pollWait(0x7f39ff5555c0, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00d8e1498, 0x72, 0x1000, 0x1019, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00d8e1480, 0xc00d806600, 0x1019, 0x1019, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00d8e1480, 0xc00d806600, 0x1019, 0x1019, 0x203001, 0x6b7480, 0xc00c5a2138)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00d636298, 0xc00d806600, 0x1019, 0x1019, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc004614cc0, 0xc00d806600, 0x1019, 0x1019, 0x22, 0x1014, 0xc00532ca48)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc00c5a2258, 0x36d1540, 0xc004614cc0, 0x415e15, 0x1b3b2a0, 0x1d9fb40)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc00c5a2000, 0x36d62a0, 0xc00d636298, 0x5, 0xc00d636298, 0x11)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00c5a2000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc00c5a2000, 0xc010274000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc00ffe2f00, 0xc004ae2658, 0x9, 0x9, 0x4779a60, 0x7f39ff6cd560, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc00ffe2f00, 0xc004ae2658, 0x9, 0x9, 0x9, 0x462713, 0x744a70ef851c0, 0xc00532ce50)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc004ae2658, 0x9, 0x9, 0x36d1280, 0xc00ffe2f00, 0x0, 0xbfa85c7300000000, 0x1c48b51db4d90, 0x47782a0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc004ae2620, 0xc00d40c600, 0xc00d40c600, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00acb2fc0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1264 +0x16f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:300 +0xd23

goroutine 5512212132 [select]:
net/http.(*http2ClientConn).roundTrip(0xc00d18b500, 0xc015013200, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:7596 +0x977
net/http.(*http2Transport).RoundTripOpt(0xc00019cf60, 0xc015013200, 0x1af0c00, 0xc000798930, 0xc00d31b380, 0x5)
	/usr/local/go/src/net/http/h2_bundle.go:6948 +0x16f
net/http.(*http2Transport).RoundTrip(...)
	/usr/local/go/src/net/http/h2_bundle.go:6909
net/http.http2noDialH2RoundTripper.RoundTrip(0xc00019cf60, 0xc015013200, 0x36d8460, 0xc00019cf60, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:9103 +0x3e
net/http.(*Transport).roundTrip(0x46af0e0, 0xc015013200, 0x1a5a980, 0xc00a76fe01, 0xc003ee0420)
	/usr/local/go/src/net/http/transport.go:515 +0xd94
net/http.(*Transport).RoundTrip(0x46af0e0, 0xc015013200, 0x1e2335d, 0xa, 0xc003ee0508)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
google.golang.org/api/transport/http.parameterTransport.RoundTrip(0xc00011cae0, 0x1e, 0x0, 0x0, 0x0, 0x0, 0x36d6340, 0x46af0e0, 0xc015013100, 0x1e3dbd9, ...)
	/work/go/pkg/mod/google.golang.org/[email protected]/transport/http/dial.go:130 +0x235
go.opencensus.io/plugin/ochttp.(*traceTransport).RoundTrip(0xc00fb56880, 0xc015013100, 0xc005019c80, 0x1, 0x1)
	/work/go/pkg/mod/[email protected]/plugin/ochttp/trace.go:84 +0x456
go.opencensus.io/plugin/ochttp.statsTransport.RoundTrip(0x36d3f60, 0xc00fb56880, 0xc015012e00, 0xc00fb5a440, 0xc00fb5a440, 0x1be9ac0)
	/work/go/pkg/mod/[email protected]/plugin/ochttp/client_stats.go:57 +0x5cc
go.opencensus.io/plugin/ochttp.(*Transport).RoundTrip(0xc000614550, 0xc015012e00, 0x0, 0x0, 0xc005256a80)
	/work/go/pkg/mod/[email protected]/plugin/ochttp/client.go:99 +0x1fb
golang.org/x/oauth2.(*Transport).RoundTrip(0xc0001116e0, 0xc015012d00, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/transport.go:55 +0x155
net/http.send(0xc015012d00, 0x36d40c0, 0xc0001116e0, 0x0, 0x0, 0x0, 0xc00f6a4190, 0xc00d31b3bb, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:252 +0x43e
net/http.(*Client).send(0xc0005529c0, 0xc015012d00, 0x0, 0x0, 0x0, 0xc00f6a4190, 0x0, 0x1, 0xc005256a80)
	/usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).do(0xc0005529c0, 0xc015012d00, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:699 +0x44a
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:567
google.golang.org/api/internal/gensupport.send(0x3716100, 0xc00f3354a0, 0xc0005529c0, 0xc015012c00, 0x47a6b80, 0xc00070fa00, 0x91fd46)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/gensupport/send.go:67 +0xef
google.golang.org/api/internal/gensupport.SendRequest(0x3716100, 0xc00f3354a0, 0xc0005529c0, 0xc015012c00, 0xc00070fd08, 0x7d, 0x7f39d51ebe40)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/gensupport/send.go:52 +0x18d
google.golang.org/api/storage/v1.(*ObjectsInsertCall).doRequest(0xc00e640410, 0x1e1b11d, 0x4, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/storage/v1/storage-gen.go:10041 +0x8cc
google.golang.org/api/storage/v1.(*ObjectsInsertCall).Do(0xc00e640410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/work/go/pkg/mod/google.golang.org/[email protected]/storage/v1/storage-gen.go:10053 +0xbd
cloud.google.com/go/storage.(*Writer).open.func1(0xc00ce54900, 0xc00f794000, 0xc00f6a4168, 0xc00d1feba0, 0x2, 0x2)
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:160 +0x79b
created by cloud.google.com/go/storage.(*Writer).open
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:110 +0x3f3

goroutine 5511303333 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f39d51ecf20, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00d1fca18, 0x72, 0x1000, 0x1018, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00d1fca00, 0xc00d9b6c00, 0x1018, 0x1018, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00d1fca00, 0xc00d9b6c00, 0x1018, 0x1018, 0x203001, 0x6b7480, 0xc003dc5638)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00f684a08, 0xc00d9b6c00, 0x1018, 0x1018, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc004f56e20, 0xc00d9b6c00, 0x1018, 0x1018, 0x1a, 0x1013, 0xc00e06aa48)
	/usr/local/go/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc003dc5758, 0x36d1540, 0xc004f56e20, 0x415e15, 0x1b3b2a0, 0x1d9fb40)
	/usr/local/go/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc003dc5500, 0x36d62a0, 0xc00f684a08, 0x5, 0xc00f684a08, 0x9)
	/usr/local/go/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc003dc5500, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc003dc5500, 0xc003bc8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc00c731b60, 0xc003fb6b98, 0x9, 0x9, 0xc00ab21180, 0x7f39ff6cce98, 0x0)
	/usr/local/go/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x36d1280, 0xc00c731b60, 0xc003fb6b98, 0x9, 0x9, 0x9, 0x942c05, 0xc0046c8660, 0xc0046c0004)
	/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc003fb6b98, 0x9, 0x9, 0x36d1280, 0xc00c731b60, 0x0, 0x0, 0xc0046c8660, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc003fb6b60, 0xc0046c8660, 0xc0046c8600, 0x0, 0x0)
	/work/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00078d6c0)
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1264 +0x16f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/work/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:300 +0xd23

goroutine 20882 [syscall, 8282 minutes]:
syscall.Syscall6(0xf7, 0x1, 0x30, 0xc00586dd08, 0x1000004, 0x0, 0x0, 0x36a0610, 0xaaaaaaaaaaaaaa, 0xc00586dd48)
	/usr/local/go/src/syscall/asm_linux_amd64.s:41 +0x5
os.(*Process).blockUntilWaitable(0xc005910000, 0x410bd2, 0xc00585a298, 0xc00585a2a8)
	/usr/local/go/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc005910000, 0xc00586de60, 0x410715, 0xc00585a240)
	/usr/local/go/src/os/exec_unix.go:22 +0x39
os.(*Process).Wait(...)
	/usr/local/go/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc005367340, 0x14, 0x0)
	/usr/local/go/src/os/exec/exec.go:507 +0x60
github.com/ShinyTrinkets/overseer.(*Cmd).run(0xc005398c60)
	/work/go/pkg/mod/github.com/maoueh/[email protected]/cmd.go:449 +0x4d9
created by github.com/ShinyTrinkets/overseer.(*Cmd).Start
	/work/go/pkg/mod/github.com/maoueh/[email protected]/cmd.go:271 +0xce

goroutine 5512207166 [select]:
github.com/dfuse-io/dhammer.(*Hammer).outputSingleBatch(0xc009aaa500, 0x37160c0, 0xc000198020, 0xc016a4d0e0, 0x1, 0x0)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:219 +0x102
github.com/dfuse-io/dhammer.(*Hammer).linearizeOutput(0xc009aaa500, 0x37160c0, 0xc000198020)
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:208 +0x1b0
created by github.com/dfuse-io/dhammer.(*Hammer).Start
	/work/go/pkg/mod/github.com/dfuse-io/[email protected]/dhammer.go:94 +0x8d

goroutine 5512212131 [select]:
cloud.google.com/go/storage.(*Writer).monitorCancel(0xc00ce54900)
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:232 +0xc7
created by cloud.google.com/go/storage.(*Writer).open
	/work/go/pkg/mod/cloud.google.com/go/[email protected]/writer.go:98 +0x333

And while I don't think it's relevant, here blocks before the crash:

Blocks before crash

info  2020-05-17T08:48:16.490 nodeos    producer_plugin.cpp:401       on_incoming_block    ] Received block 535e89aa440ff846... #121128518 @ 2020-05-17T08:48:16.500 signed by eosdotwikibp [trxs: 22, lib: 121128184, conf: 0, latency: -9 ms]
info  2020-05-17T08:48:17.075 nodeos    producer_plugin.cpp:401       on_incoming_block    ] Received block 6515612611377c1e... #121128519 @ 2020-05-17T08:48:17.000 signed by eosdotwikibp [trxs: 21, lib: 121128184, conf: 0, latency: 75 ms]
info  2020-05-17T08:48:17.077 nodeos    producer_plugin.cpp:401       on_incoming_block    ] Received block e87cfb414a6baa08... #121128520 @ 2020-05-17T08:48:17.500 signed by eosdotwikibp [trxs: 1, lib: 121128184, conf: 0, latency: -422 ms]
info  2020-05-17T08:48:18.510 nodeos    producer_plugin.cpp:401       on_incoming_block    ] Received block cbc0c3bd54815c14... #121128521 @ 2020-05-17T08:48:18.000 signed by eoseouldotio [trxs: 18, lib: 121128196, conf: 240, latency: 510 ms]

In eosq transaction page, display account(s) holding key(s) that signed transaction

One cool feature requested recently was to see the list of accounts controlling the key used to signed a transaction.

Not sure of the exact UI aspect, maybe a popup on current Signed By display line or besides each of the public key list (because there could be more than one key signing the transaction).

Some thing to note here:

We can use dfuse /v0/state/key_accounts here to achieve that.
The query must be performed using the block_num in which the transaction was, so we respect the timeframe of the transaction and the state at this block height.
There might be multiple keys to sign a transaction
There might be multiple accounts controlling the signed key, we must do the UI in consequence. It think we should aim for a design when there is a single account for the key and when there is more than that
When there is more than 1 account controlling the key, it must be made clear in the UI that it's one this account that used the key, but we don't know which one for sure

(Screenshot from https://eosq.app/tx/a4816e3a54c8f29a5c60c025bf1c6bc3c2bc11cf5dc5a9c1b4e8b979e47057a6)

When there is a single account, in the design, we could even think about showing the account straight and the key in smaller:

Signed By:   [bluebet2user](/account/bluebet2user) (<small>EOS7Dn2u2nCt7YZnC4JXEqwnMEB1oB9dyWTGHd1vmSMbhKPjCwrXC</small>)
             [bluebetproms](bluebetproms) (<small>EOS5No1zoNusvFU2yhUock7RTEkKTQeeA5gCCZV9BQi1bCzeeQPB3</small>)

(Markdown pseudo style)

dfuseeos starts too many threads by default

on a host with many CPUs, dfuse seems to want to start (almost) as many threads as there are CPUs available, where it totally doesn't need that many.

You can control this with something like export GOMAXPROCS=8, but really it shouldn't create more threads than it needs out of the box. (number of threads depends on features selected of course).

panic trying to commit a discarded txn

Seems it happened when shutting down, so perhaps a context cancellation caught, but a Commit() sent nonetheless.

Syncing mindreader with 'merge-and-upload-directly' creates a hole in continuity when it restarts

While doing phase 1 of chain sync, we use the mindreader-merge-and-upload-directly which means each time mindreader restarts, it must go back a little in time, to ensure there is no missing merge blocks bundle created.

This however is problematic until the first snapshot is taken. In this case, since there is no snapshot, mindreader will simply restart using the actual blocks.log/state file of nodeos which will create a hole in the "merged bundles". This is usually a problem only if hitting Ctrl-C. If the process is still running but nodeos restarts, I think (but not 100% sure) that information is kept in dfuse for EOSIO process and there will be no hole.

Note sure what to do here, seems it's not an easy task to cover correctly all cases. Ideally, in this "sync" mode, we should restart from scratch if we are unable to restore from snapshot, to play safe. To sure how to make that in the options directly.

A specific new option "sync"? When merge-and-upload-directly is used, it acts differently ensuring this constraints?

eosq: load config in react dev server without devProxy

We do not want to set up a devProxy server in go just for eosq, as it complicates the dev flow when not running dfuseeos.
When index.html is not templated, we will use default config values from JS code, and load API Key from environment variable.

Accessing local networks through existing tools

I'm trying to utilize Anchor Wallet by Greymass and running into issues where it won't recognize accounts on my local dfuse chain. See this conversation I had with Aaron briefly. @maoueh noted that he also ran into issues with Anchor where it was emplacing the PUB_K1 prefix.

Josh Kauffman | dfuse.io, [May 14, 2020 at 4:27:36 PM]:
Hey guys.
Trying to get anchor to function with a local network
I'm feeding it the chainID, localhost, checking the testnet checkbox. 
Going to import a private key, and it can't find the account (have tried with multiple keys that exist on the chain)
One of the other guys on my team are doing the same steps as I am on their machine, and it's finding the account. Have triple checked the steps with him, and we've done all the same steps. 
This is a dfuseeos chain if that makes a diffference (both mine and his were)

i think i see what the issue is stemming from @jestagram . 
I am passing the priv key associated to the standard EOS6MR... dev key
But in the payload being sent by Anchor, it's showing: public_key: "PUB_K1_6BtgCcdChWGARLHHfBquwMx2pwUhrnBeaaB7QPuoBGFHQYYL2L"

I think I see where the problem might be, that flag is only getting set if a auth_invalid_token_error ends up being returned, in which it retries with a API key or prompts for the key. If you had the key, it's probably not setting that flag appropriately.

We'll have to better somehow detect when it's a dfuse endpoint and set the flags in a different way.

You have any thoughts on additional ways to detect if an endpoint is a dfuse endpoint?

Support HomeBrew on MacOS

Deploying eosio and the eosio.cdt via HomeBrew has made it reasonably simple for new devs to get up and running on MacOS. It would be ideal if defuseeos followed this deployment pattern, available with a brew install dfuseeos .

Thanks.

Double slashes after hostname in URL causes all sort of weird behavior

If someone tries to hit /v1/chain API proxied calls via a POST there is a double trailing slash between the hostname and the path element of the URL, the HTTP response will be a 301 Moved Permanently```. However, returning a 301in case of HTT POST is problematic for client that respect the HTTP spec as a redirect 301 using aPOSTshould be retried using aGET` instead.

The HTTP code 307 was added exactly for this, so that client retry the request with the exact original request instead.

One would argue that it's easy to just remove the trailing slash, but sometimes it's kind of just hard to see it:

const rpc = new JsonRpc("https://kylin.eos.dfuse.io/")

In this case, when the developer will do rpc.get_block(...), the endpoint will be of the form https://kylin.eos.dfuse.io//v1/chain/get_block which is not spotted easily. For the record, it's probably the third time we got bitten by this problem.

This seems to be coming from github.com/gorilla/mux library and the StrictSlash handling that returns a 301 by default: https://godoc.org/github.com/gorilla/mux#Router.StrictSlash

We should instead send a 307 in those cases so there is no problem with POST request.

Design rework of fluxdb

State of affairs:

FluxDB returns a stream of JSON. See #18
We want streaming to use gRPC, fluxdb doesn't speak grpc/pb yet.
We want it to be slimmer: not be responsible for doing ABI serialization.
We want clear abstraction for building tables, pluggable w/ a registry.
Parallelization need to be cross-checked here with new patterns and latest code base.

@maoueh has mucho code pending to implement a few of those things.

Questions:

How do we retro-serve in eosws if we migate fluxdb to grpc? Go through GraphQL?
- How we emulate REST?
- In eosws, who would do ABI serialization?
FluxDB could live outside of dfuse-eosio, with its indexing tech and all, and EOSIO plugins would live in here.
Protobuf for certain internal data structures? Snapshot itself. Part of the registered plugin?
For Ethereum storage, we'd like to use fluxdb core: split the notion of "table" and "rows", each with the notion of block height, or chain version. All of this with support for speculative segment navigation.
- Draft what an API would look like.
How can we think of making things transaction-centric, and not block centric? Could global_sequence become the new block_num, and we could have some sort of other "rounding" feature, to simplify and shrink storage after a certain depth, but not in the reversible segment?
- Could we spin a FluxDB solely based on global_sequence?
- Could that value be a uint64 in all cases? This would almost double the size of snapshots.
  - Unless we use chain-specific snapshots protobufs ? Then we make it as we want?
Do we abstract the extraction functions, being passed a blk.Block, and returning a bunch of Rows (like in @maoueh 's branch right now).
Think of caching the snapshots + applied rows, so that next paginated query doesn't need to fetch the snapshot and the replay rows.

Nodeos config issue on exposing REST port on Docker windows

There's an issue on nodeos on exposing REST port on Docker windows that needs the IP changed from 127.0.0.1 -> 0.0.0.0

https://github.com/dfuse-io/dfuse-eosio/blob/develop/launcher/cli/config.ini.go#L48
http-server-address = 127.0.0.1:8888
To
http-server-address = 0.0.0.0:8888

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble