pftool should write MD chunk-info from single-task about marfs

[brettkettering]

This needs to be done for the Open Science MarFS version. I think this is already done, but if not it needs to be done.

[cadejager]

This was fixed by adding pre and post process option to pftool's Path.h

[jti-lanl]

Apologies. Re-opening this. What we fixed with the pre/post process was installing xattrs, etc.

I agree that post_process() seems like a natural place to add writing the N:1 chunk-info into the MD file, but it doesn't do that, yet. We need to investigate whether post_process() only gets called from a place in pftool where the whole file has been finished, or is it per-chunk (which wouldn't help), or is it per one of those block-thingies that contain many path_items (which might help, especially if they could be relied on to all be part of the same N:1 move)., and, if the latter, are the path_items all contiguous chunks of info?

Might not be too hard to resolve these questions. Then we'd know whether we can just extend post_process(), or need some other hacked Path method to be called from somewhere else in pftool.

[cadejager]

I tested this on a 100GB file while watching post_process with a print statement. The marfs chunk size was 2GB. Post process was only called once after the transfer was complete.

[brettkettering]

The end result that we need for Open Science is that a file is seen to be the correct size at the end of the file writing. This should work for PACKED, UNI, and MULTI files.

A later update can provide interim sizes as the file is being copied.

Chris D. to verify that you see the call one time for each file type (UNI and MULTI for now because PACKED is done by the script).

[cadejager]

I can confirm that on UNI and MULTI files the post_process is called once per file.

[brettkettering]

Then, I propose that you close the issue.

Thanks,
Brett

From: Chris DeJager <[email protected]mailto:[email protected]>
Reply-To: mar-file-system/marfs <[email protected]mailto:[email protected]>
Date: Monday, December 21, 2015 at 2:54 PM
To: mar-file-system/marfs <[email protected]mailto:[email protected]>
Cc: Brett Kettering <[email protected]mailto:[email protected]>
Subject: Re: [marfs] pftool should write MD chunk-info from single-task (#21)

I can confirm that on UNI and MULTI files the post_process is called once per file.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-166434529.

[brettkettering]

Open another issue for later to provide the size while the file is being copied, not just when it's done.