Comments (8)
This needs to be done for the Open Science MarFS version. I think this is already done, but if not it needs to be done.
from marfs.
This was fixed by adding pre and post process option to pftool's Path.h
from marfs.
Apologies. Re-opening this. What we fixed with the pre/post process was installing xattrs, etc.
I agree that post_process() seems like a natural place to add writing the N:1 chunk-info into the MD file, but it doesn't do that, yet. We need to investigate whether post_process() only gets called from a place in pftool where the whole file has been finished, or is it per-chunk (which wouldn't help), or is it per one of those block-thingies that contain many path_items (which might help, especially if they could be relied on to all be part of the same N:1 move)., and, if the latter, are the path_items all contiguous chunks of info?
Might not be too hard to resolve these questions. Then we'd know whether we can just extend post_process(), or need some other hacked Path method to be called from somewhere else in pftool.
from marfs.
I tested this on a 100GB file while watching post_process with a print statement. The marfs chunk size was 2GB. Post process was only called once after the transfer was complete.
from marfs.
The end result that we need for Open Science is that a file is seen to be the correct size at the end of the file writing. This should work for PACKED, UNI, and MULTI files.
A later update can provide interim sizes as the file is being copied.
Chris D. to verify that you see the call one time for each file type (UNI and MULTI for now because PACKED is done by the script).
from marfs.
I can confirm that on UNI and MULTI files the post_process is called once per file.
from marfs.
Then, I propose that you close the issue.
Thanks,
Brett
From: Chris DeJager <[email protected]mailto:[email protected]>
Reply-To: mar-file-system/marfs <[email protected]mailto:[email protected]>
Date: Monday, December 21, 2015 at 2:54 PM
To: mar-file-system/marfs <[email protected]mailto:[email protected]>
Cc: Brett Kettering <[email protected]mailto:[email protected]>
Subject: Re: [marfs] pftool should write MD chunk-info from single-task (#21)
I can confirm that on UNI and MULTI files the post_process is called once per file.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-166434529.
from marfs.
Open another issue for later to provide the size while the file is being copied, not just when it's done.
from marfs.
Related Issues (20)
- return-codes from str_2_post(), etc, should distinguish version-mismatch from other errors
- marfs_statvfs() should reflect quota-limitations HOT 1
- Installation checker HOT 1
- sync_ and close_data() failures are masked HOT 2
- clean-up: always build with DAL/MDAL HOT 1
- Multiple Simultaneous Writers HOT 6
- rename / unlink bug HOT 1
- strip out DAL/MDAL build-options HOT 1
- misconfigured RDMA auth-file gets ENODATA from fuse
- MarFS trash semantics - HOT 2
- the system calls getxattr("security.capability") on every FUSE write()! HOT 1
- marfs_rename() should do trash_truncate on the destination HOT 9
- Pftool Libxml Include Requires Subdir HOT 1
- GC Rewrite: Per-Namespace Trash
- Quota Tool Modification: Per-Namespace Trash
- MarFS should do more in the setup process HOT 1
- Create a single node MarFS example
- Move documentation for MarFS online HOT 1
- Line wrap in install doc doesn't work as expected HOT 1
- Change multi line commands in tutorial to be one command HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from marfs.