Move bad-files under volumes directory about ngas

[smclay]

Further to my previous description I got the following feedback from our ARC colleague...

• /NGASXX/bad-files needs huge storage to store files transferred from /NGASXX/volume*/staging/* when the NGAS service is restarted
• Moving files from /NGASXX/volumeXX/staging/* to /NGASXX/bad-files/ can take a long time especially when we have many large files which is what normally happens when we are mirroring files

If bad-files is located on each volume...

• No additional storage device is used
• Moving files from /NGASXX/volumex/staging/* to /NGASXX/volumeXX/bad-files/ is very fast because files are moved on the same storage device

And I can not find the reason of moving staging files to bad-files. I think 99.9% bad-files are not needed, just needs to delete frequently. Only good reason might be it can be used to resuming, I don't know how does NGAS github use bad-files.

[rtobar]

@smclay thanks for the suggestion. I agree with all the points, and I also don't see any particular reason for the bad-files area not to be turned into a per-volume directory. AFAIK bad-files is a sink, the system doesn't automatically promote any file out of it; files end up there only for inspection purposes I think, and here and there admins might want to register or locally archive bad-files.

Changes should be fairly simple, I'll try to work it out these days.

[smclay]

@rtobar thank you for taking care of this. I am sure Eisuke will be very happy as it will help solve his SAN storage mounting issues.

[rtobar]

@smclay I've implemented the changes in the issue-51 branch, with the bad-files directory now appearing under each volume directory as discussed. Unit tests have been updated to reflect this new logic. Please test at your earliest convenience and after that I'll merge back to the master.

[smclay]

@rtobar we have been testing NGAS with your latest changes. They are working well. We have not experienced any problems. Eisuke and myself appreciate your support.

[rtobar]

Great, thanks for the info! I've merged the branch now to the master.

[smclay]

@rtobar I got a message from Eisuke. He reported that "Data Check thread has scanned all the files under */volume*/ and files found under bad-files directory were reported as NOT REGISTERED FILES"
He suggests that "it may be useful for admin to know there is unprocessed files under bad-files but perhaps using another category, e.g. BAD-FILES FOUND ON STORAGE DISKS is better then NOT REGISTERED FILES
What do you think?

[rtobar]

@smclay Ahhh, that's a side effect we didn't see coming. That should not be difficult to do I think. Could you open a new issue to keep track of that separately please? Thanks!

[smclay]

OK. I create a new issue. Thanks