Main thread gets stuck randomly while awaiting for promise about quickjs_es_runtime

[andrieshiemstra]

hmm, my first guess is my ResolvableFuture isn't perfect yet.. i'll get back to you

https://github.com/HiRoFa/utils/blob/master/src/resolvable_future.rs#L32

[andrieshiemstra]

done, have fun with 0.8.2 :)

[andrieshiemstra]

ok, Ive been staring at terminals doing over a million iterations now without having the issue'..

sidenote: in utils you should match JsValueFacade::JsError { val } => Err(val), instead of the obj thing, that won't happen

in your main.rs i removed the pedning jobs call, eval handles that

//_q_js_rt.run_pending_jobs_if_any();

could you add this to your Cargo.toml

[dependencies]
...
backtrace = "0.3.56"
simple-logging = "2.0.2"
log = "0.4.17"

and this to your main()

#[tokio::main]
async fn main() -> anyhow::Result<()> {

    panic::set_hook(Box::new(|panic_info| {
        let backtrace = Backtrace::new();
        println!(
            "thread panic occurred: {}\nbacktrace: {:?}",
            panic_info, backtrace
        );
        log::error!(
                "thread panic occurred: {}\nbacktrace: {:?}",
                panic_info,
                backtrace
            );
    }));

    simple_logging::log_to_file("jstest.log", LevelFilter::Info)
        .ok()
        .expect("could not init logger");
...

That may give me some extra info if the worker thread panics.. i'll look at it again tomorrow

[andrieshiemstra]

hmm, run_pending_jobs_if_any actualy changes the order of things hapening, so I ran the test again in 10 terminals (1m iterations) but again no problems...

[andrieshiemstra]

I saw your earlier comment but it got deleted?

the 'sending on a closed channel error' could imply the promise jsvaluefacade was dropped so first i want to rule out an issue with the recursive_async crate you're using

I altered the get_as_string method like this so it does need the async-recursion annotation anymore, could you please check if your issue persists? with your updated code I still cant reproduce it, could you please share some details about your environment like linux version/rust version?

pub async fn get_as_string(
    rt: Arc<QuickJsRuntimeFacade>,
    mut val: JsValueFacade,
    reason: String,
    id: String,
) -> anyhow::Result<String, JsError> {
    println!("{} get_as_string={}", id, val.get_value_type());

    while let JsValueFacade::JsPromise { cached_promise } = val {
        let rti = rt.js_get_runtime_facade_inner().upgrade().expect("invalid state");
        let prom_res = cached_promise.js_get_promise_result(&*rti).await;
        match prom_res {
            Ok(prom_val) => {
                match prom_val {
                    Ok(prom_res_val) => {
                        val = prom_res_val
                    }
                    Err(prom_err_val) => {
                        return Err(JsError::new_string(prom_err_val.stringify()));
                    }
                }
            },
            Err(e) => {
                return Err(e);
            }
        }
    }

    match val {
        JsValueFacade::String { val } => Ok(val),
        JsValueFacade::JsObject { cached_object } => {
            let week_rti = rt.js_get_runtime_facade_inner();
            let rti = week_rti.upgrade().unwrap();
            let obj = cached_object.js_get_object(&*rti).await?;
            let stack = obj.get("stack");
            let title = obj.get("title");
            let message = obj.get("message");
            if let (Some(title), Some(message), Some(stack)) = (title, message, stack) {
                let title = if title.is_string() {
                    title.get_str()
                } else {
                    ""
                };
                let message = if message.is_string() {
                    message.get_str()
                } else {
                    "Unexpected JS Error! Please check the server log."
                };
                let stack = if stack.is_string() {
                    stack.get_str()
                } else {
                    ""
                };
                return Err(JsError::new(
                    title.to_owned(),
                    message.to_owned(),
                    stack.to_owned(),
                ));
            };
            todo!()
        }
        JsValueFacade::Undefined | JsValueFacade::Null => Ok("".to_owned()),
        JsValueFacade::JsError { val } => Err(val),
        _ => Err(JsError::new_string(format!(
            "Unexpected value found `{:?}` for {}, expected a string.",
            val, reason
        ))),
    }
}

[SreeniIO]

I saw your earlier comment but it got deleted?

Yes, I deleted it later, as it was due to a code error on my part. I missed waiting for the spawned threads to complete.

With the panic hook added, it doesn't print anything on the console when this happens.

I'm running on my Mac

I will try with the above suggestion by removing the recursive call.

[SreeniIO]

Looks like something to do with parallel threads trying to use a single runtime. When I moved runtime to it's own thread and communicated through a channel then I don't see any issue. Will do some more testing and will post my findings.

[andrieshiemstra]

If your code still hangs without the recursive annotation I would like to add some logging for debugging, does your code ever hang in debug mode?

[SreeniIO]

Yes, it does hang in debug mode as well. I've removed the recursive call.

[andrieshiemstra]

could you provide a logfile of an instance which hangs please?

with adding e.g. this to your main

simple_logging::log_to_file("jstest.log", LevelFilter::Trace).unwrap();

[SreeniIO]

Please find attached the log

jstest.log.zip

It's generated from the log3 branch
https://github.com/SreeniIO/jstest/tree/log3

[SreeniIO]

Here is another log. This got stuck sooner than the above.

jstest 2.log.zip

[SreeniIO]

Our actual software is running on Azure with Oracle Linux docker image where it get's stuck randomly. Let me setup a docker image with the above log3 branch code, so it will be easier for you to take a look.

[andrieshiemstra]

ooh after a bit of inspection it seems ResolvableFuture is far from perfect, and yes you ARE using it :) a lot

[andrieshiemstra]

i updated utils, could you test again using the latest quickjs_runtime?
e.g. Cargo.toml

quickjs_runtime = { git="https://github.com/HiRoFa/quickjs_es_runtime"}
hirofa_utils = { git="https://github.com/HiRoFa/utils"}

[andrieshiemstra]

you could get a race condition when ResolvableFuture was resolved while the other thread received but not yet added the waker

[SreeniIO]

i updated utils, could you test again using the latest quickjs_runtime?

Yeah! This worked like a charm :) Can you please release this fix and I will test it with our actual code. Thanks a lot for your help.

[SreeniIO]

you could get a race condition when ResolvableFuture was resolved while the other thread received but not yet added the waker

Indeed... it happens a lot in my case...

ƛ grep "TaskFuture::poll -> Ready @ 2" jstest.log
[00:00:01.650] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:00:16.109] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:00:19.079] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:00:19.886] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:00:29.015] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:00:54.941] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:01:01.172] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:01:05.713] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:01:09.660] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2
[00:01:10.142] (10283c580) TRACE  TaskFuture::poll -> Ready @ 2