restart is conflicted for copy vs. compare of MarFS files about pftool

[garygrider]

Overwriting anything in the dest fs on a read seems bad

---

From: Jeff Inman [email protected]
Sent: Wednesday, June 1, 2016 11:11:20 AM
To: pftool/pftool
Subject: [pftool/pftool] restart is conflicted for copy vs. compare of MarFS files (#26)

The pftool compare task uses CTM just like the copy task. During comparison, worker_comparelist() calls update_chunk() which evokes worker_update_chunk(). I think the reasoning must be that long "deep" file comparisons (i.e. with '-M') ought to be able to restart, so they should maintain CTM as they go. Furthermore, if someone attempted to compare a source with an incomplete copy (which would have copy-related CTM), the surviving CTM built during the copy task would just imply parts of the file not to compare, but the other parts (not existing) could still correctly indicate a "mismatch", when the read() on them fails. (Actually, you'd get a non-fatal read-error on each such chunk.)

The problem that crops up with chunked comparisons involving MarFS destinations is that worker_update_chunk() causes us to overwrite the chunk-info in the MDFS file. We might overwrite with correct info, but this still changes the mtime of the file, which causes MD comparisons for the remaining source/destination chunks to fail. [The destination is then re-adjusted to again have correct MD, when pftool finishes the comparison.] The upshot is that chunked comparisons of correctly copied MarFS files appear to show a "mismatch" on all but the first chunk.

The simplest solution would be to disable restarts for chunked "deep" comparisons. A better, not-much-harder approach would be to get worker_update_chunk() to know the difference between copy and compare tasks, so the call to write_chunkinfo() in libmarfs can only be called in the copy case. Another alternative would be to only do the MD comparisons when looking at the first chunk. This seems less good, because it still means messing with the MDFS file during comparisons.

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/26, or mute the threadhttps://github.com/notifications/unsubscribe/ALNxmd4rPTf6LGKboNJUY9ZGNNT53EWNks5qHb04gaJpZM4IrxgW.

[jti-lanl]

Fixed. When doing compares, we don't touch the destination. This is the "better" solution, mentioned above. There was another hitch, where update_stats() was getting called for compares. Did the same thing with that.

We still maintain CTM for compares, though. That means "deep" chunked compares can be restarted.