Restore parameters for various task counts about pftool

[garygrider]

The args for pftool in 2010 were

static struct option long_options[] =
{
{"path", required_argument, 0, 'p'},
{"copypath", required_argument, 0, 'c'},
{"resultsfile", required_argument, 0, 'r'},
{"dirproc", required_argument, 0, 'd'},
{"tapeproc", required_argument, 0, 't'},
{"ifnewer", required_argument, 0, 'i'},
{"quiet", required_argument, 0, 'q'},
{"older", required_argument, 0, 'o'},
{"younger", required_argument, 0, 'y'},
{"uid", required_argument, 0, 'u'},
{"gid", required_argument, 0, 'g'},
{"smaller", required_argument, 0, 's'},
{"bigger", required_argument, 0, 'b'},
{"mini", required_argument, 0, 'm'},
{"jumbo", required_argument, 0, 'j'},
{"Jjobid", required_argument, 0, 'J'},
{"whopping", required_argument, 0, 'w'},
{"Recurse", required_argument, 0, 'R'},
{"Maxcopysize", required_argument, 0, 'M'},
{"Zerodev", required_argument, 0, 'Z'},
{"Testmode", required_argument, 0, 'T'},
{"smallcopyblockksize", required_argument, 0, 'z'},
{"Waitidletape", required_argument, 0, 'W'},
{"Database", required_argument, 0, 'D'},
{"AhostDatabase", required_argument, 0, 'A'},
{"UserDatabase", required_argument, 0, 'U'},
{"elfpoolsize", required_argument, 0, 'e'},
{"Elfpool", required_argument, 0, 'E'},
{"largepoolsize", required_argument, 0, 'l'},
{"Largepool", required_argument, 0, 'L'},
{"Xlargepool", required_argument, 0, 'X'},
{"Prettyoutput", required_argument, 0, 'P'},
{"Optimizepanfslayout", required_argument, 0, 'O'},
{"Nosplit", required_argument, 0, 'N'},
{"Inputlist", required_argument, 0, 'I'},
{"Serverdmapi", required_argument, 0, 'S'},
{"Fallbackstoragepool", required_argument, 0, 'F'},
{"Chunkdirs", required_argument, 0, 'C'},
{"fusedir", required_argument, 0, 'f'},
{"Knock", required_argument, 0, 'K'}, // for tape recall testing HB CHEN 0212-2009
{"Bcompare", required_argument, 0, 'B'}, // for compare
{"Qcomapre", required_argument, 0, 'Q'}, // for compare
{"help", no_argument, 0, 'h'},
{0,0,0,0}
};

And then

/nprocs must be greater than 1 manager + 1 output + nreaddirprocs + ntapeprocs/
if (nproc < (2 + nreaddirproc + ntapeproc + 1 )) {

and
/*assign the different ranks the jobs they will do 0-manager, 1-output,
the readdirprocs starts at 2, the tape procs starts at 2+nreaddirprocs,
and the stat/copy slaves start at 2+nreaddirprocs+ntapeprocs */

rank zero was manager, rank one was output then the next set of procs were readdir procs, then the next set of procs were tape recall procs, then the remainder were stat and copy procs.

If you go back even further I am quite sure there was a way to control number of stat procs too, but I think Aaron felt like stat was highly related to copy, copy cant move forward without stat info.

Anyway, once upon a time…

For now maybe we can see if 4 procs hangs, even though we wont know why it doesn’t hang, maybe we can limp with 4 or even 5 procs.

From: Brett Kettering [mailto:[email protected]]
Sent: Thursday, March 24, 2016 11:26 AM
To: pftool/pftool
Subject: [pftool] Restore parameters for various task counts (#10)

There used to be command-line parameters for readdir, stat, and copy task counts. This allowed the user to specify how much parallelism you wanted in each of stat, readdir, and copy because you want to tune those separately.

Comb through the code and see if there are variables for these or if there is a way to make variables for each of these separate from the mpirank or thread counts in the current pftool.cfg files.

Gary Grider feels it's very important to be able to set these three things independent of one another.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/10

[brettkettering]

This will be more important with sharded directories. Not critical for Secure Campaign.

[cadejager]

Adding dedicated stat processes would indeed be more difficult because that is not currently a separate task.

Without too much trouble I can allow certain processes to be dedicated to directory, take, or copy work. I propose having a parameter for each of these so that the total number for processes must be > 3 + dir_proc_count + copy_proc_count + tape_proc_count. Any remaining processes would maintain the current behavior where they could be used for any task.

[cadejager]

It is now possible to limit the number of stat processes with -D.