Shall we have built-in progress bar support for r_session? about callr

[gaborcsardi]

cc @lionel @hadley @jimhester

[hadley]

In terms of sharing a progress bar from the child back to the parent? I think that would be great.

[gaborcsardi]

@hadley I was wondering if it should be here or in some higher level parallel package. But yeah, probably makes sense to have it here.

[gaborcsardi]

Rough plan:

Implementation notes

• The function would throw conditions, and the injected code will catch them, and send them to the
  parent process. This way progress events can be generated the same way for local and remote
  processes.
• Ultimately we want to throttle the thrown conditions, i.e. do not throw more than one in a 200ms
  interval, except for urgent events, like termination. Maybe we want to throttle by type, e.g. progress
  update, status update, estimate update, etc.
• We need to have a C API, but that will be different, because you cannot easily throw conditions
  from C. So we'll just write to the pipe from C. The C API will need to be throttled as well.
• We need to work out how loops with progress bars can be embedded into each other. Maybe
  progress bars need ids, and the conditions will contain the ids.

The API

Not decided yet how the conditions will be generated, but they could roughly correspond to these functions:

• add_job(name, status, total, format, estimate, auto_estimate)
• set_job_progress(id, progress)
• add_job_progress(id, increment)
• set_job_status(id, status)
• set_job_estimate(id, seconds)
• add_job_output(id, output, type)
• complete_job(id, succeeded, output, error)

[jimhester]

FWIW there was a project about parallel progress bars at this years' unconf (https://github.com/ropenscilabs/jobstatus), maybe not really that useful

Throwing conditions from C isn't really that hard, I think it would be best to use the same API for both cases, e.g. if you think the pipe interface is better we should provide R functions to wrap it.

[gaborcsardi]

@jimhester re C API, I am just worried that for C code throwing a condition is too expensive.

In the end the progress event will be written to a pipe, maybe the C API can just write it w/o a throwing a condition....

[gaborcsardi]

OK, callr now catches conditions of class callr_message in the subprocess, transmits them (the whole object) to the parent process, and r_session$run() re-throws them in the parent process.

So, as long as the progress is signalled via messages, callr does everything it needs to do, to support progress bars from child processes. Of course some new APIs are still needed in the progress package: r-lib/progress#72