timelydataflow / diagnostics Goto Github PK
View Code? Open in Web Editor NEWDiagnostic tools for timely dataflow computations
License: MIT License
Diagnostic tools for timely dataflow computations
License: MIT License
Hi,
I have tried to use this tool with different examples and crate versions.
It always gets stuck: pressing enter has no result.
To reproduce:
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y cargo git
RUN git clone https://github.com/TimelyDataflow/timely-dataflow
WORKDIR timely-dataflow
RUN cargo install tdiag
ENV PATH="$PATH:/root/.cargo/bin"
Run:
docker build -t foo .
And run this docker in two different shells:
docker run --name foo_container --rm -it foo tdiag --source-peers 2 graph --out graph.html
docker exec -it foo_container env TIMELY_WORKER_LOG_ADDR="127.0.0.1:51317" cargo run --example pagerank 1000 1000000 -w 2
Differential just got TimelyDataflow/differential-dataflow#201, which makes it possible to track batches created in transient trances. Seems like differential arrangements
could do something useful with that.
Hello. I was trying to follow the example on the README.md but tdiag gets stuck.
cargo run --release -- --source-peers 2 graph --out graph.html
env TIMELY_WORKER_LOG_ADDR="127.0.0.1:51317" cargo run --release --example pagerank 1000 100000 -w 2
Listening for 2 connections on 127.0.0.1:51317
Trace sources connected
Press enter to generate graph (this will crash the source computation if it hasn't terminated).
Looking at the stack trace of tdiag there are two threads. The main thread is waiting on a thread join. Thread2 also seems stuck on await_events
. Stack trace for Thread2:
futex_wait_cancelable 0x00007ffff7d9c376
__pthread_cond_wait_common 0x00007ffff7d9c376
__pthread_cond_wait 0x00007ffff7d9c376
std::sys::unix::condvar::Condvar::wait condvar.rs:73
std::sys_common::condvar::Condvar::wait condvar.rs:50
std::sync::condvar::Condvar::wait condvar.rs:200
std::thread::park mod.rs:923
<timely_communication::allocator::thread::Thread as timely_communication::allocator::Allocate>::await_events thread.rs:44
<timely_communication::allocator::generic::Generic as timely_communication::allocator::Allocate>::await_events generic.rs:99
timely::worker::Worker<A>::step_or_park worker.rs:216
timely::execute::execute::{{closure}} execute.rs:206
timely_communication::initialize::initialize_from::{{closure}} initialize.rs:269
std::sys_common::backtrace::__rust_begin_short_backtrace backtrace.rs:130
std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} mod.rs:475
<std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once panic.rs:318
std::panicking::try::do_call panicking.rs:297
__rust_try 0x000055555661a74d
std::panicking::try panicking.rs:274
std::panic::catch_unwind panic.rs:394
std::thread::Builder::spawn_unchecked::{{closure}} mod.rs:474
core::ops::function::FnOnce::call_once{{vtable-shim}} function.rs:232
<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once boxed.rs:1034
<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once boxed.rs:1034
std::sys::unix::thread::Thread::new::thread_start thread.rs:87
start_thread 0x00007ffff7d95609
clone 0x00007ffff7ed1103
Please lmk if I missed something when executing the commands?
Named regions, named scopes and named operator invocations don't seem to be shown on the graph, which greatly limits how useful the data it gives is
This may be needed for complex graphs like Frank's epic doop graph.
Various snippets from #1:
I've been thinking a bit about how to present these, and one thought was: maybe it makes sense to have two nodes for the feedback node, and to not connect them other than visually. This maybe allows the graph to dangle a bit better, and reveals the acyclic definitions.
I think this could be a good use case for a touch of interactivity, e.g. draw the nodes somewhat differently to indicate an outgoing / incoming feedback edge, and then highlight the pair when the user hovers on either of the two nodes.
As an experiment, I built an extra script for adding DataScript into the mix. This is intended to be completely opt-in, without changing anything about the current representation.
I also added a hook to re-render the whole thing reactively.
This should give us a low-overhead (no React!) way to experiment with a few more dynamic features, such as highlighting feedback edges.
It would be helpful to have scopes be exported as well, which would allow us to do things such as collapsing / expanding scopes.
The crates.io release currently points at timely & differential 0.9, so it can't be used together with a 0.10 codebase.
It is hard to diagnose a "stuck" timely dataflow computation, where for some reason there is a capability (or perhaps message) in the system that prevents forward progress. In the system there is fairly clear information (in the progress tracking) about which pointstamps have non-zero accumulation, and although perhaps not strictly speaking a "visualization" we could imagine extracting and presenting this information.
@antiguru recently had a similar issue, in which he wanted to "complete" a dataflow without simply exiting the worker (to take some measurements), and when he attempts this the dataflow never reports completion. The root cause was ultimately that a forgotten input was left un-closed.
One idiom that seemed helpful here was to imagine a version of the dataflow graph that reports e.g. whether operators have been tombstoned or not (closed completely, memory reclaimed). This would reveal who was keeping a dataflow open, which is a rougher version of what is holding a dataflow back. We might also look for similar idioms that allow people to ask, for a given timestamp/frontier, which operators have moved past that frontier and which have not, revealing where in the dataflow graph a time is "stuck".
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.