Progress indicator about pftool

[jti-lanl]

Good idea. Just to be clear, it seems like we'd want the periodic update to show the new total of data_moved, the cumulative avg BW, as well as the same stats for the given period alone.

On May 4, 2016, at 10:21 AM, David Bonnie [email protected]
wrote:

It would be super helpful to have a basic progress indicator short of verbose mode, which makes it waaay too hard to see any errors flying by. Even as someone familiar with the tool, you can't be quite sure at runtime that anything is getting done (from a single terminal) without being blasted with millions of lines of output.

So, I'm proposing something similar to:

dbonnie@testsystem: pfcp /intputfiledir /outputdir
[regular pftool startup output]
Status: W files stat'd, X files moved, Y MB moved, XX:YY elapsed
(repeat every few minutes or something, nothing fancy, no overwriting/animations)
[regular pftool final ouput]

Something simple that gives users a good idea that the program is still at least doing something for them.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

[thewacokid]

Yes, exactly. I'm not sure if bandwidth over the whole run vs. the last period is critical, but if it's easy, sure! I'm fairly sure all of this information is readily available and should be a pretty quick patch.

EDIT: Also, to be clear, I think this should be the default output mode.

[garygrider]

Not sure how you do this for a "given period of time" since pftool (the output proc) is event driven.
I would suggest options for -status 100 every 100 ops or -status 1000 every 1000 ops or something, so its not time driven,
since we don't have the concept of time really, just events.

[thewacokid]

So I guess we want to do event based per X events OR Y seconds/minutes since last even (when a new event comes in) just so people still get some kind of progress indicator even if things are slow.

That'd be great to start, and we can rate limit the output if we want to get fancy once it's working.

[brettkettering]

From #23
From Gary Grider:

One of the things we learned in Open Science was that users run pfcp non verbose and then kill it because of lack of feedback. Then they run pfcp verbose and hate the deluge. Then they kill it and run it again without verbose.

Need a lighter weight verbose. Perhaps multiple "v"s, e.g., -v, -vv, -vvv, -vvvv with the last being the current verbose mode. Or, -v with an argument. The argument is the number of files to pass between status update and no argument is the current super verbose.

Users are interested in progress of the copy. So, the number of files inspected, moved, etc. Every 100 files, every 1000 files etc. Simply let the trailer proc count and write status to stderr or out. Something easy to grep. Trivial but nice to have.

[jti-lanl]

We're already using -v[v[v[v]]] to indicate different levels of diagnostics/behavior that are handy for debugging. Those could be moved to a different option, to make way for the more-user-oriented notion of verbosity.

[cadejager]

We already have support for multiple levels of verbosity I can demote some things to a lower (more Vs) level of verbosity or would just the progress indicator by without and verbose flag be enough.

[jti-lanl]

Seems like our more-advanced -v options are really more of debugging support. They should probably just move to a separate "debug" option, like -d, -dd, -ddd, etc. Then, -vv etc could do what Gary suggests.

[garygrider]

Something like that seems good

---

From: Jeff Inman [email protected]
Sent: Wednesday, May 11, 2016 10:23:35 AM
To: pftool/pftool
Cc: Grider, Gary Alan; Comment
Subject: Re: [pftool/pftool] Progress indicator (#19)

Seems like our more-advanced -v options are really more of debugging support. They should probably just move to a separate "debug" option, like -d, -dd, -ddd, etc. Then, -vv etc could do what Gary suggests.

You are receiving this because you commented.
Reply to this email directly or view it on GitHubhttps://github.com//issues/19#issuecomment-218512414

[brettkettering]

We need this in order for users not to get impatient with transfers and wonder if anything is happening. Jeff stated that he has a plan for implementing a feedback mechanism that is not too verbose and communicates progress at a reasonable rate for a human watching the process.

[jti-lanl]

I re-jiggered the command-line options, as suggested above. They now look like this:

  -v    show old-style per-chunk-or-file output
  -vv   also show old-style 'stat' diagnostics

  -g    do the "spinning waiting for gdb attach" [formerly -vvvv]
  -gg   show detailed per-transaction curl diagnostics [formerly -vvv]

If no '-v' flags are specified, we now only print a line of cumulative output every 10 seconds. For now, looks like this: (copying an 11GB file, in this example)

[normal "HEADER" output]
INFO ACCUM files/chunks: 2 data: 2.0 GB / 11.0 GB avg BW: 204.4 MB/s errs: 0
INFO ACCUM files/chunks: 4 data: 4.0 GB / 11.0 GB avg BW: 204.3 MB/s errs: 0
INFO ACCUM files/chunks: 6 data: 6.0 GB / 11.0 GB avg BW: 203.9 MB/s errs: 0
INFO ACCUM files/chunks: 8 data: 8.0 GB / 11.0 GB avg BW: 203.6 MB/s errs: 0
INFO ACCUM files/chunks: 9 data: 9.0 GB / 11.0 GB avg BW: 183.2 MB/s errs: 0
[normal "FOOTER" output]

In this example, "11.0 GB" is the stat size of a large file being copied in chunks. You can tell you're making progress because (a) the number of files/chunks is changing, and (b) the amount of data moved is increasing. If you were not using chunking, you'd still have (b).

If there's a lot of read-directory/stat going on, but no file-work, you will see increases in the total size (denominator) for the "data" portion, like this:

INFO ACCUM files/chunks: 0 data: 0 B / 3.7 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 4.5 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 5.1 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 5.6 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 5.9 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 6.2 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 6.7 MB avg BW: 0 B/s errs: 0

The "avg BW" is average over the entire run, up to that point. It's just based on the number of completed chunks (no deep knowledge of not-yet-reported work form the workers). Therefore, it may bounce around a little for smaller transfers, or in the early parts of large transfers. Seems to be in line with the final avg BW reported in the footer.

Errors will always print, such as mismatches in a recursive directory-compare.

File-compares work "correctly" in the non-verbose format, but all you'll see is the number of errors to tell you there were some mismatches.

RESTARTS
The "files/chunks" tells you how many files and/or chunks were moved or compared, which is not necessarily the same as the indices of those files or chunks. Similarly, the amount of data moved doesn't include the amount of data skipped, because process_stat_buffer() doesn't count that. Therefore, if you run a restart on a chunked file (e.g. pftool -w 0 -n), you might get the mistaken impression that it is doing the same work that it did before. This is awkward to fix. Probably, process_stat_buffer() should maintain "skipped" stats and report to master, so we can adjust the numbers.

You can always add a '-v', to double-check that it is using offsets that indicate a restart. Then quit and resume the job without -v.

[garygrider]

Nice, some feedback will hopefully help the users not be so baffled

From: Jeff Inman <[email protected]mailto:[email protected]>
Reply-To: pftool/pftool <[email protected]mailto:[email protected]>
Date: Tuesday, June 14, 2016 at 8:15 AM
To: pftool/pftool <[email protected]mailto:[email protected]>
Cc: First name Last name <[email protected]mailto:[email protected]>, Comment <[email protected]mailto:[email protected]>
Subject: Re: [pftool/pftool] Progress indicator (#19)

I re-jiggered the command-line options, as suggested above. They now look like this:

-v show old-style per-chunk-or-file output
-vv also show old-style 'stat' diagnostics

-g do the "spinning waiting for gdb attach" [formerly -vvvv]
-gg show detailed per-transaction curl diagnostics [formerly -vvv]

If no '-v' flags are specified, we now only print a line of cumulative output every 10 seconds. For now, looks like this: (copying an 11GB file, in this example)

[normal "HEADER" output]
INFO ACCUM files/chunks: 2 data: 2.0 GB / 11.0 GB avg BW: 204.4 MB/s errs: 0
INFO ACCUM files/chunks: 4 data: 4.0 GB / 11.0 GB avg BW: 204.3 MB/s errs: 0
INFO ACCUM files/chunks: 6 data: 6.0 GB / 11.0 GB avg BW: 203.9 MB/s errs: 0
INFO ACCUM files/chunks: 8 data: 8.0 GB / 11.0 GB avg BW: 203.6 MB/s errs: 0
INFO ACCUM files/chunks: 9 data: 9.0 GB / 11.0 GB avg BW: 183.2 MB/s errs: 0
[normal "FOOTER" output]

In this example, "11.0 GB" is the stat size of a large file being copied in chunks. You can tell you're making progress because (a) the number of files/chunks is changing, and (b) the amount of data moved is increasing. If you were not using chunking, you'd still have (b).

If there's a lot of read-directory/stat going on, but no file-work, you will see increases in the total size (denominator) for the "data" portion, like this:

INFO ACCUM files/chunks: 0 data: 0 B / 3.7 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 4.5 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 5.1 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 5.6 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 5.9 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 6.2 MB avg BW: 0 B/s errs: 0
INFO ACCUM files/chunks: 0 data: 0 B / 6.7 MB avg BW: 0 B/s errs: 0

The "avg BW" is average over the entire run, up to that point. It's just based on the number of completed chunks (no deep knowledge of not-yet-reported work form the workers). Therefore, it may bounce around a little for smaller transfers, or in the early parts of large transfers. Seems to be in line with the final avg BW reported in the footer.

Errors will always print, such as mismatches in a recursive directory-compare.

File-compares work "correctly" in the non-verbose format, but all you'll see is the number of errors to tell you there were some mismatches.

RESTARTS
The "files/chunks" tells you how many files and/or chunks were moved or compared, which is not necessarily the same as the indices of those files or chunks. Similarly, the amount of data moved doesn't include the amount of data skipped, because process_stat_buffer() doesn't count that. Therefore, if you run a restart on a chunked file (e.g. pftool -w 0 -n), you might get the mistaken impression that it is doing the same work that it did before. This is awkward to fix. Probably, process_stat_buffer() should maintain "skipped" stats and report to master, so we can adjust the numbers.

You can always add a '-v', to double-check that it is using offsets that indicate a restart. Then quit and resume the job without -v.

You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/19#issuecomment-225894727, or mute the threadhttps://github.com/notifications/unsubscribe/ALNxmQbmO0GFyxoZAPgaoQSZkmOApXySks5qLreQgaJpZM4IXUZ6.

[jti-lanl]

Need some more support to make it clear what's going on during a restart, and/or long metadata crawling period. Make sure users will see the stat-ing activity in the form of the denom in the "data" field in the periodic output, in the case of many small files.

Add tracking of "skipped" files (due to CTM), in process_stat_buffer(). Report this to master, where it can then be added into the data-moved (numerator) area, and the files/chunkks count, but don't include it in the BW.

[brettkettering]

In today's discussion with Gary, it was decided that there should be some sort of indicator that we're stating files to check to see how much needs to be operated on and an indicator of action on how much. So, metadata action and data action. It doesn't need to be perfect, like accounting for stuff moved in previous commands where the current command is continuing the moving, etc. That can come later. For the Secure Campaign Release we want some level of feedback that the users can see something is happening in the metadata work and the data work.

[brettkettering]

Right now the denominator increases as the stat'ing in going on to show how much needs to be examined. The numerator increases once data movement begins.

There's no adjustment to the numerator to account for a restart where the stat'ing determines how much has already been moved and does not need to be moved again.

For Secure Campaign, it's good enough to give the user the comfort of knowing that the job is progressing. Later we'll come back to this and make an enhancement to account for restarts where some data has already been moved and is not moved again.

[cadejager]

Commit 8e91074 make it such that status now includes previously copied bytes.