bastion-rs / bastion Goto Github PK

Highly-available Distributed Fault-tolerant Runtime

License: Apache License 2.0

Rust 99.78% Shell 0.22%

actor asynchronous concurrency distributed distributed-systems fault-tolerance hacktoberfest numa-aware rust rust-language smp supervision

bastion's People

Contributors

Stargazers

Watchers

Forkers

cjschneider2 isgasho 0xflotus paulyc preachermanx baitcenter xmclark johntitor tchigher mh84 cptrodgers marulitua rust-stuff minijackson opencollective jangocheng onsails rtyler harryzhangau zmilan shaneutt deontologician backwardn damienstanton neuroradiology enkore mbrukman sgnls anhntbk08 zeta1999 mdheller fossabot fabjan attila-lin cksac lukemathwalker keruspe federicoponzi asevans48 chaosstudygroup alexbosy yutiansut linecode anlumo creativcoder strogo ninja73 nicolaiunrein tuxiqae rustyforks daocuong surechen scrabsha isirfanm rustbunker davidyew jancajthaml fosdickio simonsan therustmonk samsplunks doytsujin baajarmeh kanru icodein cjfuller cbonaudo randomicon00 duzhanyuan eliasyaoyc drvspw ives9638 zhouyao1605 dherse pavelzx zhaopufeng zhangxi123051 iamdylanngo dylanops cometyang tko-cactus qm3ster mapbased kotlin2018 smartbigant q741242673 fritiofhedstrom 3exp8 crt-fork bonedaddy kitcatier younghakim7 codeoverflow aleksandergondek shenzhongshan wdshin suddenlyhazel 78tuba317 llaith-oss shabbirhasan1

bastion's Issues

Macro style meta-architecture instantiation

Instead of writing a bunch of boilerplate code. People can able to do this:

supervisor! { OneForAll,
    child! {
        |context, msg: Box<dyn Message>| {
            // some thing over here
        }
    }

    children! { 100,
        |context, msg: Box<dyn Message>| {
            // some thing over here too
        }
    }
}

Max retries implementation

Implementation for max retries is needed. Error delegation should take action with respect to max retries. This will minimize error chaining.

Pass last message and panic error to the "before_restart" and "after_restart" callbacks

Hey 👋!

We should pass the eventual message that was received by a Child before faulting to the before_restart callbacks and the eventual panic error to the after_restart callbacks.

After panic callbacks should be called without polling

Hey 👋!

Currently, for an after panic callback to get called, the RecoverableHandle associated with it needs to get polled even though the proc might have already panicked.

This would save us from periodically polling handles in bastion's Children.

Jettison global run queue in executor

Work should be directly distributed to pinned run queues and we don't need a global run queue. This is a residual problem.

We need to remove this and replace it with processes that have their own priorities to schedule beforehand.

Priority will be determined based on reductions and API's directive.

Bastion is looking for maintainers!

As I have mentioned in last paragraph on Reddit, Bastion has a very bright future and there is room for it to improve and become a prominent runtime.

We are looking for maintainers who will:

Simplify the code and guide the development
Develop networking between cross installations.
Make this project reactor-independent.
Groom issues
Develop new features
Make documentation and infrastructure shine

If you are interested in these topics above and want to deep dive into Rust. This is a good start.
This project and other crates that will be developed independently with this project will reside under Bastion RS.

Looking forward to hearing from future maintainers!

Support configuring the system

Hey 👋!

We, at least, need to support allowing to disable the panic message catching and configuring what #77 might use.

Docs and debug impls for 0.3.0

Write docs and debug implementations for:
bastion, bastion-executor and lightproc.

Put this at the crate root.

// Force missing implementations
#![warn(missing_docs)]
#![warn(missing_debug_implementations)]

Shutdown implementation

Program recovery might be managed programmatically in addition to signals.

Add "receive" like method to message decoding

Potential executor simplification

Tokio is exploring Structured Concurrency Support. If that's informed by and ultimately accommodates the needs of Bastion its custom executor could potentially be removed.

Make msg! macro to return value

msg! macro returning value is useful for constructing async interfaces to actors:

struct AsyncFoo {
    child: ChildRef,
}

impl AsyncFoo {
    pub async fn open_bar(&self, open_bar: &OpenBar) -> Result<BarId, Error> {
        let answer = self
            .child
            .ask(open_bar)
            .expect("Failed to send message");

        msg! { answer.await.expect("Failed to receive answer"),
            msg: Result<BarId, Error> => {
                msg;
            };
            _: _ => Err(format_err!("wrong answer"));
        }
    }
}

Currently, this code leads to compilation error, compiler sees () return type but expects Result<BarId, Error>.

Same problem happens with explicit return inside a macro:

struct AsyncFoo {
    child: ChildRef,
}

impl AsyncFoo {
    pub async fn open_bar(&self, open_bar: &OpenBar) -> Result<BarId, Error> {
        let answer = self
            .child
            .ask(open_bar)
            .expect("Failed to send message");

        msg! { answer.await.expect("Failed to receive answer"),
            msg: Result<BarId, Error> => {
                return msg;
            };
            _: _ => return Err(format_err!("wrong answer"));
        }
    }
}

Context spawn impl

Enable the system to use context as children generation and scoped execution.
Should be done before scoped API implementation takes place.

After panic callbacks should know the error

Hey 👋!

It would be better if panic callbacks had access to the catched error. Something like:

Fn(Box<dyn Any + Send + 'static>)

100% CPU usage

Hi,

First of all: your project seems really neat and I'm on my way trying to create my first application using it :-)

Having that said:

Version: 0.3
Platform: Linux (kernel 5.3.8), Intel x86-64
Subsystem: bastion-executor
Compiler: Rust nightly (so it's 1.41.0-nightly (e87a205c2 2019-11-27) now)

Running any of the provided examples (or just even plainly invoking Bastion::init()) makes my CPU usage immediately jump to 100% (on a single core); the examples themselves are working fine, it's just that I wouldn't expect such CPU load from an application that's basically idling.

perf told me that 99% of the time is spent inside the load balancing thread:

    pub fn sample() {
        thread::Builder::new()
            .name("load-balancer-thread".to_string())
            .spawn(move || {
                loop {
                    if let Ok(mut stats) = load_balancer::stats().try_write() {
                        // Write latest downscaled mean to statistics
                        stats.mean_level = stats
                            .smp_queues
                            .values()
                            .sum::<usize>()
                            .wrapping_div(placement::get_core_ids().unwrap().len()); // <- the `get_core_ids()` is the actual hot spot here
                    }

                    // Try sleeping for a while to wait
                    thread::sleep(Duration::new(0, 10));
                    // Yield immediately back to os so we can advance in workers
                    thread::yield_now();
                }
            })
            .expect("load-balancer couldn't start");
    }

... and so I've come up with two changes that reduce CPU usage to 10%[1]:

Since all we're doing here is placement::get_core_ids().unwrap().len(), I've created placement::count_core_ids() function that returns number of active cores:

pub fn count_core_ids() -> usize {
    let mut count = 0;

    if let Some(full_set) = get_affinity_mask() {
        for i in 0..CPU_SETSIZE as usize {
            if unsafe { CPU_ISSET(i, &full_set) } {
                count += 1;
            }
        }
    }

    count
}

I've changed the thread::sleep(Duration::new(0, 10)); into thread::sleep(Duration::from_millis(1));.

I'm not sure if my changes are correct correct, but at least they brought the CPU usage down for now.

So I guess my question says: is the high CPU usage an expected behaviour? If so, what can I do to avoid it?

Thanks,
Patryk.

[1] which, to be fair, is still kinda unsatisfactory, because all that time is "wasted" counting active cores

Use scoped threads if there is no panic catch implemented inside for micro concurrency

No example from https://bastion.rs working

Hi!
None of the examples from https://bastion.rs/#one is working for me. I create a common issue here because I think it is no coincidence that both are broken. One does not compile, the other give a run-time error.

Common

Cargo.toml:

[package]
name = "hello-world"
version = "0.1.0"
authors = ["Stefano Probst <[email protected]>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
fort = "0.2.0"
bastion = "0.2.0"

Software versions:

$ cargo version
cargo 1.39.0 (1c6ec66d5 2019-09-30)

$ echo $PWD
/tmp/hello-world

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 19.10
Release:	19.10
Codename:	eoan

$ uname -srv
Linux 5.3.0-19-generic #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019

Example "configure every piece by yourself"

Code:

use bastion::prelude::*;

fn main() {
    Bastion::platform();

    // Define, calculate or prepare messages to be sent to the processes. 
    let message = String::from("Some message to be passed");

    Bastion::spawn(
        |context: BastionContext, msg: Box<dyn Message>| {
            // Message can be selected with receiver here. Take action!
            receive! { msg,
                String => |e| { println!("Received string :: {}", e)},
                i32 => |e| {println!("Received i32 :: {}", e)},
                _ => println!("No message as expected. Default")
            }

            // Do some processing in body
            println!("root supervisor - spawn_at_root - 1");

            // Rebind to the system
            context.hook();
        },
        message,
    );

    Bastion::start()
}

Error messages:

$ RUST_BACKTRACE=1 cargo run
   Compiling hello-world v0.1.0 (/tmp/hello-world)
error[E0433]: failed to resolve: use of undeclared type or module `objekt`
  --> src/main.rs:12:13
   |
12 | /             receive! { msg,
13 | |                 String => |e| { println!("Received string :: {}", e)},
14 | |                 i32 => |e| {println!("Received i32 :: {}", e)},
15 | |                 _ => println!("No message as expected. Default")
16 | |             }
   | |_____________^ use of undeclared type or module `objekt`
   |
   = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
error: could not compile `hello-world`.

To learn more, run the command again with --verbose.

Example "fort the proc-macro"

#[fort::root]
fn main() {
    println!("Running in Bastion runtime!");
}

Run-time Error message:

Long backtrace inside

$ RUST_BACKTRACE=1 cargo run 2> error.log
[break code via ctrl+c]
$ head -n 180 error.log
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/hello-world`
thread 'tokio-runtime-worker-0' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:378:21
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:76
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:60
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1030
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1412
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:64
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:196
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:210
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:473
  11: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:380
  12: rust_begin_unwind
             at src/libstd/panicking.rs:307
  13: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  14: core::panicking::panic
             at src/libcore/panicking.rs:49
  15: core::option::Option<T>::unwrap
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libcore/macros.rs:12
  16: bastion::context::BastionContext::spawn
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/bastion-0.2.0/src/context.rs:213
  17: hello_world::main::{{closure}}
             at src/main.rs:1
  18: <alloc::boxed::Box<F> as core::ops::function::Fn<A>>::call
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/liballoc/boxed.rs:936
  19: bastion::bastion::Bastion::spawn::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/bastion-0.2.0/src/bastion.rs:401
  20: futures::future::lazy::Lazy<F,R>::get
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/lazy.rs:64
  21: <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/lazy.rs:82
  22: futures::future::catch_unwind::<impl futures::future::Future for std::panic::AssertUnwindSafe<F>>::poll
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/catch_unwind.rs:49
  23: <futures::future::catch_unwind::CatchUnwind<F> as futures::future::Future>::poll::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/catch_unwind.rs:32
  24: std::panicking::try::do_call
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panicking.rs:292
  25: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  26: std::panicking::try
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panicking.rs:271
  27: std::panic::catch_unwind
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panic.rs:394
  28: <futures::future::catch_unwind::CatchUnwind<F> as futures::future::Future>::poll
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/catch_unwind.rs:32
  29: futures::future::chain::Chain<A,B,C>::poll
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/chain.rs:26
  30: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/then.rs:32
  31: <alloc::boxed::Box<F> as futures::future::Future>::poll
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/future/mod.rs:113
  32: futures::task_impl::Spawn<T>::poll_future_notify::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329
  33: futures::task_impl::Spawn<T>::enter::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399
  34: futures::task_impl::std::set
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/std/mod.rs:83
  35: futures::task_impl::Spawn<T>::enter
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:399
  36: futures::task_impl::Spawn<T>::poll_fn_notify
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:291
  37: futures::task_impl::Spawn<T>::poll_future_notify
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.29/src/task_impl/mod.rs:329
  38: tokio_threadpool::task::Task::run::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/task/mod.rs:145
  39: core::ops::function::FnOnce::call_once
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libcore/ops/function.rs:227
  40: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panic.rs:315
  41: std::panicking::try::do_call
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panicking.rs:292
  42: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  43: std::panicking::try
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panicking.rs:271
  44: std::panic::catch_unwind
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panic.rs:394
  45: tokio_threadpool::task::Task::run
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/task/mod.rs:130
  46: tokio_threadpool::worker::Worker::run_task2
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/worker/mod.rs:567
  47: tokio_threadpool::worker::Worker::run_task
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/worker/mod.rs:459
  48: tokio_threadpool::worker::Worker::try_run_owned_task
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/worker/mod.rs:390
  49: tokio_threadpool::worker::Worker::try_run_task
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/worker/mod.rs:297
  50: tokio_threadpool::worker::Worker::run
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/worker/mod.rs:241
  51: tokio::runtime::threadpool::builder::Builder::build::{{closure}}::{{closure}}::{{closure}}::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/threadpool/builder.rs:390
  52: tokio_timer::timer::handle::with_default::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/handle.rs:101
  53: std::thread::local::LocalKey<T>::try_with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:262
  54: std::thread::local::LocalKey<T>::with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:239
  55: tokio_timer::timer::handle::with_default
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/timer/handle.rs:84
  56: tokio::runtime::threadpool::builder::Builder::build::{{closure}}::{{closure}}::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/threadpool/builder.rs:382
  57: tokio_timer::clock::clock::with_default::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/clock/clock.rs:137
  58: std::thread::local::LocalKey<T>::try_with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:262
  59: std::thread::local::LocalKey<T>::with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:239
  60: tokio_timer::clock::clock::with_default
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.11/src/clock/clock.rs:117
  61: tokio::runtime::threadpool::builder::Builder::build::{{closure}}::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/threadpool/builder.rs:381
  62: tokio_reactor::with_default::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.10/src/lib.rs:237
  63: std::thread::local::LocalKey<T>::try_with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:262
  64: std::thread::local::LocalKey<T>::with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:239
  65: tokio_reactor::with_default
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.10/src/lib.rs:217
  66: tokio::runtime::threadpool::builder::Builder::build::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.22/src/runtime/threadpool/builder.rs:380
  67: tokio_threadpool::callback::Callback::call
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/callback.rs:22
  68: tokio_threadpool::worker::Worker::do_run::{{closure}}::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/worker/mod.rs:127
  69: tokio_executor::global::with_default::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.8/src/global.rs:209
  70: std::thread::local::LocalKey<T>::try_with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:262
  71: std::thread::local::LocalKey<T>::with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:239
  72: tokio_executor::global::with_default
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.8/src/global.rs:178
  73: tokio_threadpool::worker::Worker::do_run::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/worker/mod.rs:125
  74: std::thread::local::LocalKey<T>::try_with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:262
  75: std::thread::local::LocalKey<T>::with
             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/thread/local.rs:239
  76: tokio_threadpool::worker::Worker::do_run
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/worker/mod.rs:116
  77: tokio_threadpool::pool::Pool::spawn_thread::{{closure}}
             at /home/stefano/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.16/src/pool/mod.rs:345
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[BASTION] - [ERROR] - Panic happened in root level child - Any
thread 'tokio-runtime-worker-0' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:378:21
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:76
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:60
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1030
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1412
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:64
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49

Conclusion

I have no idea if this two things are connected as this was litaraly my first try of bastion/fort.
The demo project from above can be found underhello-world.zip. I deleted the target folder because otherwies this would be a few hundred megabyte.

Support for hot code swapping

Describe the solution you'd like
For systems that are intended to never stop, it's helpful if components of running systems, or perhaps additionally entire systems proper, may be updated gracefully.

If components are updated, associated message queues should not lose messages. What other guarantees should be expected?

Would it be beneficial to add support for hot code swapping via WebAssembly-based components until Rust has a [more] stable ABI and Rust-based support becomes more practical?

Original comment "HCS (hot-code-swap)" by @vertexclique at https://lobste.rs/s/mhky0b/announcing_bastion_0_3_highly_available#c_u1w0jg

Explore using github actions

Maintaining appveyor + travis is quite a bit painful. Github Actions can be a solution. If it is feasible let's move to that.

CONTRIBUTING.md link in README doesn't work

The link to the CONTRIBUTING.md file in the README.md doesn't work.
It still points to the CONTRIBUTING.md file in this repository, but it was moved to here

Proptest and concurrency tests for the executor

bastion-executor needs these tests. Find out a way to organize testing.

Process stack needs to store data

We want to carry actual data with process stack but we can't because of size bounds are varying. We need to find a way to do this to make simpler callback implementation.

pub struct ProcStack<S> {
    pub state: S,

    pub before_start: Option<Arc<dyn Fn(&mut S) + Send + Sync>>,
    // ...
}

Question: project communication?

Does bastion have a public facing IM or forum for discussion? If not, should we make one?

Feature request: custom panic handling

Attaching a custom panic handler would be useful in applications that want to run logging or cleanup code on catastrophic failures.

The panic handler shouldn’t interfere with the rusts built in panic handling solution.

Support for tracing

Describe the solution you'd like
Could bastion be improved by adding support for tracing?

Related issues: #2, #77

Executor development and design

The new executor design document is here:
https://paper.dropbox.com/doc/Bastion-Executor-Design-Document-AmsdBId9x5CDqL5ewJT0IroAg-ged3kQPsVFNOIwuhgkeIY

This is a bigger feature that needs heavy collab from both runtime and also proc side.

Main requirements

Fair scheduling with multiqueue
Optimal performance distribution with minimal cache misses
Thread recovery

Dynamic supervision

looking for impls out there
message queue impl to take over the replacements
replacement syntax and definition of replacement conditions
complete branch migration

windows and stable support

Does this project aim to support stable channel and windows targets?

I was unable to build bastion on stable-x86_64-pc-windows-gnu. There is one unix-only package and some required nightly features.

I created a draft PR that removes the unused nightly features and swaps the sigint catching for a platform solution using ctrlc crate.

I tested build and test on a few targets:

stable-x86_64-pc-windows-gnu
nightly-x86_64-pc-windows-gnu
stable-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu

It all seems to work. Is this project interested in taking on support for stable and windows?

Reword README.md (for all workspace projects)

Reword README so people should understand what Bastion does and what is the point of it.

Identify messages sender and send messages back

Describe the solution you'd like
Add a way to identify a message's sender and/or get a reference to it to send messages back.

Move parking_lot to std

Sooner or later this will happen so move back to std Mutex.

It is super easy and nice for newcomers to this project.

Configurable system tick

Write a configurable system tick:

if possible separately configurable for any supervisor
if possible separately configurable for any children group
prevent thundering herd on close

Link and Monitor reflection to Runtime

Stale copy from rkallos's comment on lobste.rs:

This looks pretty cool! I definitely see the resemblance to Erlang’s model of lightweight processes. That said, I couldn’t find the primitives upon which Erlang’s supervisors are built; links and monitors.

The idea of uni- or bi-directional links between processes seems to be more flexible and powerful than representing supervisor trees directly in code. Two examples come to mind:

Process A waiting for a response from process B, but process B crashed after receiving the message from A. This could be solved by having process A create a uni-directional link (‘monitor’ in Erlang) to B before sending the message.

Process A is responsible for maintaining state about a set of other processes managed by one or more supervisors. With links in Erlang, this is easy, and with trap_exit, process A can see how these processes exit, and update its state accordingly.

With supervision trees, you gain the ability to manage how some groups of processes are managed. With links and monitors, you gain the building blocks on which Erlang’s supervision trees are built, and much more.

Observer implementation for seeing the runtime status

There should be a UI interface to see what is going on inside the system. That will be a dashboard.
Right from the Erlang:

Considerations

This can be feature gateable or not.
Can be another crate under a workplace.

Message gating and natural replication over the broadcast

Message gating for the destination – aka routes known messages.
Natural replication or broadcast needed instead of the ordinary round-robin.

Fix clippy errors coming from lightproc

Blocking pool for the system

We want to incorporate a blocking pool to handle blocking operations separately from the SMP parallelism. Since SMP parallelism is affine and prone to congestion during the blocking operations we should dedicate blocking pool.

One important thing is here that yielding a thread in a blocking pool shouldn't be based on durations like any other Rust runtime implementation. It will be based on ß-reductions. ß-reductions will be explained later during the implementation phase.

Existing blocking pool implementations can be used(thou I suggest to incorporate initial implementation linked below). A separate global queue should be used to dispatch to the pool.

Roadmap:

Implement initial adaptive blocking pool design
Implement methods to spawn blocking code.
Implement macro/s to spawn blocking code inside children.
Count reductions to yield threads back.
Yield threads when it is needed.
Profit!

Re-add logs

Hey 👋!

We need to re-add logs and the display of backtraces (in our own way).

Global/Local State API

We need to have a storage backend to share data between processes (Global)
Share data in the same children group (Local)

The uses of `unsafe`

It's unlikely that a project described as "Highly-available Distributed Fault-tolerant Runtime" would prefer to use unsafe. May I ask therefore what criteria its use has within the bastion project?

I have an interest in potentially using bastion as well in understanding the uses (code) of unsafe in the evolution of Rust. (language)

Worker thread recovery for the executor

The executor needs to recover worker threads on internal worker thread panic.

Currently, we continue operation with the rest of the threads that are mapped onto the cores. The panicked thread should be assigned to the same affinity that it left.

This case is very edgy but needed for tolerance.

Add a "spawn" method to "Bastion"

Describe the solution you'd like
Add a spawn method to Bastion that would roughly translate to Bastion::children(|ch| ch.with_exec(_));.

Concurrency tests for whole runtime

use loom or alternatives. Needs research.

Development of lightweight process implementation

~~Task architecture should be refactored to use async-task.~~
Lightproc is created to have more suitable task system for Bastion (or any other supervision-based system on Rust)

Use futures::future::Abortable to implement global Kill strategies

Replace all future spawn principles to use:
https://docs.rs/futures-preview/0.3.0-alpha.18/futures/future/struct.Abortable.html

That will make the system hook much more reliable. Because we won't wait for exiting from the future if we want to kill a future.

Allow use of answer! outside of msg!

Describe the solution you'd like
Create a new global macro answer!(msg, your_answer); and update the answer!(your_answer); created by calls to msg!(...); to support it.

Documentation for the docs.rs

Write documentation for every piece.
redraw the readme's architecture diagram. Clearly.

Get notified when a message has been received

Describe the solution you'd like
Rename Answer to Ask and create two new structs called Tell (returned by tell) and Broadcast (returned by broadcast). The two new structs will implement Future<Output = ()> and become ready when the message has been received by all the targeted children.

Children group elements launched before the group itself

Hey 👋!

Currently, to allow ChildrenRef to both contain a list of a ChildRefs and be returned by Bastion::children and Supervisor::children_ref, the children's elements are launched before it (the Childs are launched by the creating method while the Children is launched by the Supervisor that received the Message::Deploy containing it).

We could either:

Continue doing that.
Remove the elems method from ChildrenRef or replace it to return the number of elements instead of a list of references.
Make Bastion::children and Supervisor::children_ref launch the Children themselves and send an handle to the Supervisor instead of the children group.

bastion-rs / bastion Goto Github PK

bastion's People

Contributors

Stargazers

Watchers

Forkers

bastion's Issues

Common

Example "configure every piece by yourself"

Example "fort the proc-macro"

Conclusion

Main requirements

Considerations

Recommend Projects

Recommend Topics

Recommend Org

Jobs