GithubHelp home page GithubHelp logo

Comments (2)

gbartolome-avax avatar gbartolome-avax commented on August 13, 2024 1

Per Slack conversation with @aaronbuchwald Data Platform would archive HyperSDK payloads using a similar pattern we use now when we ingest our EVM based subnets into our Data Lake

Chain Ingestion - On-Chain Producer

It will depend on a subscriber based producer/consumer push pattern.

  1. parent consumer will subscribe to HyperSDK messages and push messages to a messaging stream, in this case Kafka
  2. child consumers will subscribe to HyperSDK Kafka topic for all payloads

from hypersdk.

aaronbuchwald avatar aaronbuchwald commented on August 13, 2024

Linking a few relevant issues here:

Export Block/Tx/State Diffs to External Store

#961

I think the best way to support block/tx indexing is with this accepted subscriber pattern. Exporting state diffs would be a change from the current interface that we could optionally support if needed. Depending on the VM and use case, this may push a lot of data, so I'd prefer to export state diffs to an external store if the need arises rather than prioritizing it and changing the code to support it now.

If this is completed w/o exporting state diffs to an external store, we should open a new GitHub issue for state diffs as a potential future improvement.

Support S3 Archiver

#531
#697

This would be great to support. To avoid feature bloat in the HyperSDK, I'd prefer an S3 archiver implementation to be implemented as an external service to the HyperSDK using the gRPC Accepted Subscriber.

from hypersdk.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.