pfcm appears to use "real" stat through FUSE - about pftool

[garygrider]

Sounds like someone called stat instead of Marfs stat
Assume in the copy where it is verifying dest

If it hasn't changed from long ago

I suppose we could start fuse everywhere for now on the batch nodes

Maybe someone missed an indirection level

---

From: David Bonnie [email protected]
Sent: Monday, March 28, 2016 9:42:05 AM
To: pftool/pftool
Subject: [pftool] pfcm appears to use "real" stat through FUSE - (#14)

Multi-host pfcm jobs appear to only compare properly on the node with FUSE mounted for MarFS:

(output shortened for brevity, the only rank on the FUSE host is RANK 5):
RANK 9: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150809.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150809.0 -- MISSING DESTINATION
RANK 7: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150723.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150723.0 -- MISSING DESTINATION
RANK 6: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150712.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150712.0 -- MISSING DESTINATION
RANK 3: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150714.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150714.0 -- MISSING DESTINATION
RANK 5: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20151030.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20151030.0 -- SUCCESS
INFO FOOTER ====== NONFATAL ERRORS = 167

INFO FOOTER Total Files/Links Examined: 211
INFO FOOTER Total Dirs Examined: 2
INFO FOOTER Total Files Compared: 211
INFO FOOTER Elapsed Time: 0 seconds

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/14

[thewacokid]

These do only pop up for compare jobs, so apparently it takes a different path through the code than copy jobs. Copy jobs do resolve stats properly through the MarFS library.

[jti-lanl]

Yep, compare_file() apparently hasn't gotten any of the marfs love that went into copy_file().

It looks like everything is just straight POSIX ops, including open(). So, if two files happen to match in all their stat attributes, the compare will move on to comparing the contents of a POSIX file against the contents of the MDFS file.

There are only a few operations in compare_file(), so it should be pretty easy to marfs-ize it: PathFactory::create_shallow() with source and dest, then use stat/open/close methods.

Also, the hardwired comparison of stat attributes should be replaced with a call to samefile(), so that once we fix that, it will work everywhere.

Jeff

On Mar 28, 2016, at 9:47 AM, garygrider [email protected]
wrote:

Sounds like someone called stat instead of Marfs stat
Assume in the copy where it is verifying dest

If it hasn't changed from long ago

I suppose we could start fuse everywhere for now on the batch nodes

Maybe someone missed an indirection level

---

From: David Bonnie [email protected]
Sent: Monday, March 28, 2016 9:42:05 AM
To: pftool/pftool
Subject: [pftool] pfcm appears to use "real" stat through FUSE - (#14)

Multi-host pfcm jobs appear to only compare properly on the node with FUSE mounted for MarFS:

(output shortened for brevity, the only rank on the FUSE host is RANK 5):
RANK 9: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150809.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150809.0 -- MISSING DESTINATION
RANK 7: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150723.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150723.0 -- MISSING DESTINATION
RANK 6: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150712.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150712.0 -- MISSING DESTINATION
RANK 3: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150714.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150714.0 -- MISSING DESTINATION
RANK 5: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20151030.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20151030.0 -- SUCCESS
INFO FOOTER ====== NONFATAL ERRORS = 167

INFO FOOTER Total Files/Links Examined: 211
INFO FOOTER Total Dirs Examined: 2
INFO FOOTER Total Files Compared: 211
INFO FOOTER Elapsed Time: 0 seconds

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/14
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

[jti-lanl]

compare_file() was always TBD.

On Mar 28, 2016, at 9:56 AM, Jeff Inman [email protected]
wrote:

Yep, compare_file() apparently hasn't gotten any of the marfs love that went into copy_file().

It looks like everything is just straight POSIX ops, including open(). So, if two files happen to match in all their stat attributes, the compare will move on to comparing the contents of a POSIX file against the contents of the MDFS file.

There are only a few operations in compare_file(), so it should be pretty easy to marfs-ize it: PathFactory::create_shallow() with source and dest, then use stat/open/close methods.

Also, the hardwired comparison of stat attributes should be replaced with a call to samefile(), so that once we fix that, it will work everywhere.

Jeff

On Mar 28, 2016, at 9:47 AM, garygrider [email protected]
wrote:

Sounds like someone called stat instead of Marfs stat
Assume in the copy where it is verifying dest

If it hasn't changed from long ago

I suppose we could start fuse everywhere for now on the batch nodes

Maybe someone missed an indirection level

---

From: David Bonnie [email protected]
Sent: Monday, March 28, 2016 9:42:05 AM
To: pftool/pftool
Subject: [pftool] pfcm appears to use "real" stat through FUSE - (#14)

Multi-host pfcm jobs appear to only compare properly on the node with FUSE mounted for MarFS:

(output shortened for brevity, the only rank on the FUSE host is RANK 5):
RANK 9: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150809.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150809.0 -- MISSING DESTINATION
RANK 7: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150723.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150723.0 -- MISSING DESTINATION
RANK 6: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150712.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150712.0 -- MISSING DESTINATION
RANK 3: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20150714.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20150714.0 -- MISSING DESTINATION
RANK 5: INFO DATACOMPARE compared /lustre/scratch5/dbonnie/errorlists/archive.errorlist.20151030.0 to /campaign/admins/dbonnie/errorlists/archive.errorlist.20151030.0 -- SUCCESS
INFO FOOTER ====== NONFATAL ERRORS = 167

INFO FOOTER Total Files/Links Examined: 211
INFO FOOTER Total Dirs Examined: 2
INFO FOOTER Total Files Compared: 211
INFO FOOTER Elapsed Time: 0 seconds

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/14
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

[jti-lanl]

Fixed. (Pending push.) Tested with pftool copy of a few 4k files, then pftool compare, on machines having a marfs mount as well as machines without.

Compare of file-contents (pftool -M) also works on these small files.

TBD: There is untested support, using the same mechanisms that work in copy_file(), to deal with N:1 files (i.e. comparing the actual contents of files with '-M').

[jti-lanl]

Done. Verified 'pftool -w 2 -M' on an N:1 file.

There was also another bug with COMPARE/LIST jobs, where pftool was causing the destination to become unreadable. That's fixed, too.