ipvm-wg / homestar

Homestar is the individual node that makes up the Everywhere Computer network (similar to how IPFS Kubo, Iroh, Nabu, or other implementation nodes make up the IPFS network). It's written in Rust and is designed for performance and composability.

Homestar

Outline

• Quickstart
• Packages
• Running Examples
• Workspace
• Contributing
• Releases and Builds
• Getting Help
• External Resources
• License

Quickstart

If you're looking to help develop Homestar, please dive right into our development guide.

Otherwise, the easiest way to get started and see Homestar in action is to follow-along with our examples. To start, try out our image-processing websocket relay example, which integrates Homestar with a browser application to run a statically-configured workflow. The associated README.md walks through what to install (i.e. rust, node/npm, ipfs), what commands to run, and embeds a video demonstrating its usage.

Throughout the Homestar ecosystem and documentation, we'll draw a distinction between the host runtime and the support for different guest languages and bindings.

If you're mainly interested in learning how to write and build-out Wasm components (currently focused on authoring in Rust), please jump into our homestar-functions directory and check out our examples there.

Packages

Each Homestar release will also build packages for distribution across different platforms.

• homebrew: brew install fission-codes/fission/homestar This includes ipfs in the install by default.
• npm: npm install homestar-runtime -g Wraps the homestar-runtime binary in a node script.

Running Examples

All examples contain instructions for running them, including what to install and how to run them. Please clone this repo, and get started!

Each example showcases something specific and interesting about Homestar as a system.

Our current list includes:

• websocket relay - An example (browser-based) application that connects to the homestar-runtime over a WebSocket connection in order to run a couple static Wasm-based, image processing workflows that chain inputs and outputs.

Workspace

This repository is comprised of a few library packages and a library/binary that represents the Homestar runtime. We recommend diving into each package's own README.md for more information when available.

Core Crates

• homestar-invocation

  The invocation library implements much of the Ucan Invocation specification and is used as the foundation for other packages in this workspace and within the runtime engine.

• homestar-wasm

  This wasm library manages the wasmtime runtime, provides the IPLD to/from WIT interpreter/translation-layer, and implements the input interface for working with Ipvm's standard Wasm tasks.

  You can find the spec for translating between IPLD and WIT runtime values based on WIT types here.

  Note: This library also includes a WIT world for bindings implemented on the host runtime.

• homestar-workflow

  The workflow library implements workflow-centric Ipvm features and is used as the foundation for other packages in this workspace and within the runtime engine.

Runtime Crate

• homestar-runtime

  The runtime is responsible for bootstrapping and running nodes, scheduling and executing workflows as well as tasks within workflows, handling retries and failure modes, etc.

Non-published Crates

• homestar-functions/*

  homestar-functions is a directory of helper, test, and example crates for writing and compiling Wasm component modules using wit-bindgen.

  You can find our component packages on wa.dev.

• homestar-schemas

homestar-schemas is a crate for generating OpenRPC docs and JSON Schemas that document the homestar-runtime JSON-RPC API, workflows, and receipts.

• examples/*

  examples contains examples and demos showcasing Homestar packages and the Homestar runtime. Each example is set up as its own crate, demonstrating the necessary dependencies and setup(s).

Contributing

🎈 We're thankful for any feedback and help in improving our project! We have a focused development guide, as well as a more general contributing guide to help you get involved. We always adhere to our Code of Conduct.

Releases and Builds

Crates, Tags, and GitHub Releases

Homestar uses release-plz to publish crates, tags, changelogs, and GitHub Releases. Upon merging, a release-plz bot PR, four crates are continuously published, all tied to the same cargo version currently (though this may change in the future):

• homestar-runtime
• homestar-invocation
• homestar-workflow
• homestar-wasm

Build Targets

Every GitHub release of the homestar-runtime contains build assets for running the homestar-runtime on different target architectures, as well as DEB and RPM packages (tagged with the architectured they were compiled for). Our homebrew package for the runtime is also tied to releases and can be installed with brew install fission-codes/fission/homestar.

We also leverage cross for locally cross-compiling to varying Linux and Apple target platforms.

NPM Packages

We also release some of our cross-compiled runtime binaries as npm binary packages:

• homestar-runtime - This is the main package that installs the os specific binary package and runs it.
• homestar-darwin-arm64
• homestar-darwin-x64
• homestar-linux-arm64
• homestar-linux-x64
• homestar-windows-x64

Getting Help

For usage questions, usecases, or issues reach out to us in our Discord channel.

We would be happy to try to answer your question or try opening a new issue on GitHub.

External Resources

• What Is An IPVM
• IPVM: High-Level Spec
• Contributing Research
• Seamless Services for an Open World by Brooklyn Zelenka
• Foundations for Open-World Compute by Zeeshan Lakhani
• IPVM: The Long-Fabled Execution Layer by Brooklyn Zelenka
• IPVM - IPFS and WASM by Brooklyn Zelenka
• Breaking Down the Interplanetary Virtual Machine
• Ucan Invocation Spec

License

This project is licensed under the Apache License 2.0, or http://www.apache.org/licenses/LICENSE-2.0.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

homestar's Issues

Support expressive input types

In the current proof of concept, we only have support for i32s. This needs to be generalized to a much broader set of types, at least those available in IPLD. This both needs a clear ABI (e.g. C conventions) and/or support for the Wasm component model & WIT.

Bidirectional IPLD from/to WIT interpreter

Summary

IPLD <=> WIT interpreter for executing functions within Wasm components.

Flexible Types (not part of IPLD proper)

• record(s)
• flag(s)
• option(s)
• result(s)
• variant(s)
• enum(s)
• union(s) (unions have been 🪓'ed!)
• tuple(s)

ETA: 2023-10-20

Link/CID AOT Validation

Summary

For deterministic wasm, we should verify all links as part of the ahead-of-time "compilation" of workflows before any execution takes place. We should fail the workflow and return errors for all CIDs that do not exist at the time of execution.

If links do exist on the network but are not local, we should stream in the necessary (IPLD) blocks over the network to make the execution feasible, which is related to #78.

This does not involve the coordination piece of who should actually run the workflow, but only at the point when a node is deemed ready to execute it.

RUSTSEC-2021-0059: `aesni` has been merged into the `aes` crate

aesni has been merged into the aes crate

Details
Status	unmaintained
Package	aesni
Version	0.10.0
URL	RustCrypto/block-ciphers#200
Date	2021-04-29

Please use the aes crate going forward. The new repository location is at:

<https://github.com/RustCrypto/block-ciphers/tree/master/aes>

AES-NI is now autodetected at runtime on i686/x86-64 platforms.
If AES-NI is not present, the aes crate will fallback to a constant-time
portable software implementation.

To prevent this fallback (and have absence of AES-NI result in an illegal
instruction crash instead), continue to pass the same RUSTFLAGS which were
previously required for the aesni crate to compile:

RUSTFLAGS=-Ctarget-feature=+aes,+ssse3

See advisory page for additional details.

Split out homestar-core (workflows, etc) from networking/runtime.

Workflow Properties / Invariants

Note:
Relates to:

• ipvm-wg/spec#26

ETA: 2023-12-31

Integration: IPVM alongside Kubo

Summary

TBD

Components

• TBD

ETA: 2024-02-19

Open World: Registration of capabilities and task types

Summary

RUSTSEC-2021-0064: `cpuid-bool` has been renamed to `cpufeatures`

cpuid-bool has been renamed to cpufeatures

Details
Status	unmaintained
Package	cpuid-bool
Version	0.2.0
URL	RustCrypto/utils#381
Date	2021-05-06

Please use the `cpufeatures`` crate going forward:

<https://github.com/RustCrypto/utils/tree/master/cpufeatures>

There will be no further releases of cpuid-bool.

See advisory page for additional details.

Matchmaking

Dependencies

• This task has to be completed after #84 and #88

Note:
Relates to ipvm-wg/spec#24

ETA: 2023-12-31

Compute: Include & Index resolved values within Task Invocation

Summary

"Instruction" Invocations are content-addressed as given, which can/will typically include promise pipelined CIDs that must be resolved during execution. However, once resolved (before the actual execution of the function itself), we don't currently have a way to equate the resolved instruction (i.e. resolved with values) with the one given CIDs initially.

Essentially, the instructions given CIDs should be treated the same as those substituted with values from the output of those other task/instruction CIDs.

This will involve tracking pre/post-deferred instruction Cids for lookup on receipt checking.

Node Bootstrapping: Sharing information for coordination

Summary

Nodes need to bootstrap with information and share that information with other homestar nodes in a peer group. This opens up distributed scheduling and coordination.

Components

• What to capture for the capability matrix per node:
  • System metrics I: (available) CPU, memory, disk, etc
  • System metrics II: (available) GPU
• Gossiping information to a subset of peers, a "peer group", on a windowed interval.
• TBD

ETA: 2024-02-19

RUSTSEC-2022-0081: json is unmaintained

json is unmaintained

Details
Status	unmaintained
Package	json
Version	0.12.4
URL	maciejhirsz/json-rust#205
Date	2022-02-01

Last release was almost 3 years ago.

The maintainer is unresponsive with outstanding issues.

One of the outstanding issues include a possible soundness issue.

Possible Alternative(s)

The below list has not been vetted in any way and may or may not contain alternatives;

• serde_json
• json-deserializer
• simd-json

See advisory page for additional details.

Expand guest-wasm crate with docs and few more examples

Metering

Summary

We want to be able to calculate and properly meter fuel & memory usage on wasm functions within a workflow, dependent on workflow settings.

Goals

• Test and showcase fuel decreasing over a computation workflow
• Test and showcase memory decreasing over a computation workflow

ETA: 2023-09-30

Fuel-use

ETA: 2023-10-30

Workflow Parser

Summary

For the MVP, Workflows will be rendered as JSON and need to be loading, parsed, and then executed---really queued for execution. Eventually, we'll allow for other input formats, and #103 for generating workflow configuration and translation.

Goals

• Make sure workflows are convertible to JSON and back.
• Run multiple workflows over CLI.
• Load workflows from CLI .json files.

ETA: 2023-08-15

Move from old spec to current spec: 0.1.0

Summary

We need to move to the current spec view on a Task and work with sub-tasks vs batching.

https://github.com/ucan-wg/invocation

1st-party Effects: HTTP (GET, POST, PUT)

Summary

We want to have a first-party effect (i.e. Wasm/Wasi) for HTTP calls.

Components

• TBD

ETA: 2024-02-19

Better abstractions for working with libp2p/gossipsub

Completed IPFS and libp2p Integration

ETA: 2023-08-15

RUSTSEC-2020-0056: stdweb is unmaintained

stdweb is unmaintained

Details
Status	unmaintained
Package	stdweb
Version	0.4.20
URL	koute/stdweb#403
Date	2020-05-04

The author of the stdweb crate is unresponsive.

Maintained alternatives:

• wasm-bindgen
• js-sys
• web-sys

See advisory page for additional details.

impl gossipsub for receipt topic pubsub layer

Summary

Use gossipsub (and mdns) to share receipts (possibly other data) with connected peers.

Affinities

Dependencies

• This task has to be completed after #84 and before #89

ETA: 2023-12-31

wasmer/wasm value into ipld

Scheduler: Parallel vs Sequential Execution of Workflow Tasks

ETA: 2023-08-15

Create default task specs (IPFS-flavored Wasm, HTTP, DNS)

Summary

• Include Task Types

Problem / Need / Opportunity

ETA: AAAA-MM-DD

WASI Component for streaming in blocks from IPFS

ETA: 2023-11-03

3rd-party Host Integrations

Summary

We'd like to support first-class interop with serverless/edge-compute functions within a workflow, particularly these two "types of tasks":

• AWS Lambda
• Fastly Compute@Edge

Incorporation details are still TBD.

Captured Receipts

ETA: 2023-12-31

Executor for JS hosts / JS Runtime

Summary

Browser (host) support via (jco)

Components

• TBD

ETA: 2024-05-14

Task Permissions: (UCAN validation) on workflows and tasks

Summary

We want to leverage Ucans to authorize and validate task invocations and workflow execution. The current Ucan invocation spec has this at the task level, but we'd direct this information from workflows at the top level, pushing information into individual tasks, and use it at the receipt level as well.

Components

• TBD

ETA: 2023-12-20

RUSTSEC-2020-0168: mach is unmaintained

mach is unmaintained

Details
Status	unmaintained
Package	mach
Version	0.3.2
URL	fitzgen/mach#63
Date	2020-07-14

Last release was almost 4 years ago.

Maintainer(s) seem to be completely unreachable.

Possible Alternative(s)

These may or may not be suitable alternatives and have not been vetted in any way;

• mach2 - direct fork

See advisory page for additional details.

Less Anyhow, More ThisError for library crates

Summary

Runtime should rely on anyhow, but the library packages (e.g. core, wasm), should leverage only thiserror, with specific error information and names.

Checklist

• homestar-core
• homestar-wasm

Discovery

Dependency

• Discovery has to be completed before #88 and #89

ETA: 2023-11-03

Integration tests with wasm components

Distributed Runtime + Scheduler

Summary

• Failure Handling, Traces
• Virtual Resiliency
  • a mechanism in a (possibly distributed) programming and
    execution environment, typically employing a log, which exploits the replayably
    deterministic nature and serializilbility of an application to automatically mask
    failure.

ETA: 2023-12-31

Payment Channel Integration

Summary

Problem / Need / Opportunity

Success Scenario

Notes

Relates to:

• #100
• #101

ETA: 2024-03-31

Plugin Substrate to Extend Runtime

Summary

Problem / Need / Opportunity

Success Scenario

ETA: 2024-03-31

JS Wasm-based Browser Library/Extension

Summary

Problem / Need / Opportunity

Success Scenario

ETA: 2023-06-30

Task Properties/Invariants

ETA: 2023-12-31

Self-hosted version/deployment

Summary

We want to be able to run a Homestar (runtime) node and deploy it as a binary and a Docker container.

Goals

• Docker build (homestar/server start)
• *-nix binary
• Deployment workflows/examples

ETA: 2023-09-30

Add dependency info of `libsqlite3` in README

Summary

Tests require libsqlite3 to be installed.

Problem

Was an unexpected build failure on a new install.

Desktop (please complete the following information):

• OS: debian 11

Open World: Handling Bad Actors

Summary

Networking: Compute Performance, Probes

Summary

TBD

RUSTSEC-2021-0060: `aes-soft` has been merged into the `aes` crate

aes-soft has been merged into the aes crate

Details
Status	unmaintained
Package	aes-soft
Version	0.6.4
URL	RustCrypto/block-ciphers#200
Date	2021-04-29

Please use the aes crate going forward. The new repository location is at:

<https://github.com/RustCrypto/block-ciphers/tree/master/aes>

AES-NI is now autodetected at runtime on i686/x86-64 platforms.
If AES-NI is not present, the aes crate will fallback to a constant-time
portable software implementation.

To force the use of a constant-time portable implementation on these platforms,
even if AES-NI is available, use the new force-soft feature of the aes
crate to disable autodetection.

See advisory page for additional details.

IPLD conversion as Async Tap

Incremental Receipt Reuse

Be able to re-use receipts for incremental computation within a workflow.

ETA: 2023-08-15

Cost Semantics for Execution

Summary

DSLs / Tooling for writing tasks and using workflows

Summary

The current MVP version will run workflows via configuration, i.e. JSON, YAML, etc. We need to investigate and build a suite of libraries and tools to generate workflows for composing tasks, task-types, etc.

ETA: 2024-03-31