Support for overlong filenames about mutagen

[xenoscopic]

Sorry for the slow reply. I'll have a look into this ASAP. It was my understanding that the Go runtime handled overly long paths automatically, but perhaps that's only on Windows. What platform are you seeing this error on? Linux?

[philippgadow]

Hi Jacob,
thanks for the response and looking into this! This issue surely is not of high priority as most users will use sensible pathnames.
This error is happening on Linux, to be more specific Ubuntu 16.04.

[xenoscopic]

Can you tell me the length of the filename as well as the length of the complete path? There are some hard limits in Linux that (as far as I know) you can't get around (255 characters for filename and 4096 characters for paths). Some of these are configurable, and some are limited by the actual file system in use.

[philippgadow]

Sure,
the error report looks like this

Beta:
        URL: /home/pgadow/coding/atlas/monoV/MonoVHFramework-00-26-01-local
        Status: Connected
        Problems:
                CxAODMaker/data/efficiency.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_e24_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.LooseAndBLayerLLH_d0z0_v11_isolLooseTrackOnly.NEW.root: unable to ensure path does not exist: unable to determine path existence: lstat /home/pgadow/coding/atlas/monoV/MonoVHFramework-00-26-01-local/CxAODMaker/data/efficiency.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_e24_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.LooseAndBLayerLLH_d0z0_v11_isolLooseTrackOnly.NEW.root: file name too long
                CxAODMaker/data/efficiencySF.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_e24_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.LooseAndBLayerLLH_d0z0_v11_isolLooseTrackOnly.NEW.root: unable to ensure path does not exist: unable to determine path existence: lstat /home/pgadow/coding/atlas/monoV/MonoVHFramework-00-26-01-local/CxAODMaker/data/efficiencySF.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_e24_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.LooseAndBLayerLLH_d0z0_v11_isolLooseTrackOnly.NEW.root: file name too long

[xenoscopic]

That's a strange one. All components of those paths seem like they should be supported, even with ATLAS' crazy naming schemes. So I'm thinking the problem is one of a few things (in approximately decreasing order of likelihood):

• PATH_MAX or NAME_MAX on your system are not the standard 4096/255 (respectively). I'm guessing you're on SL 6 or 7? I guess I could see CERN doing something weird like that. You could check the Linux headers and see (look in linux/limits.h - not sure where this lives on RHEL-based systems).
• Some length limitation that Go's runtime is enforcing when making the stat/lstat system call. I can dig in and look at this, but I've looked at that code many times and never seen anything.
• Some limitation of the stat/lstat system call on Linux that restricts the full path to something like 255 characters. Quite unlikely - I think it should accept up to PATH_MAX characters.
• Some component of the path you're using is a symlink that resolves to a REALLY long path (> 4096 characters in length). You would probably know if this is the case.
• Somehow there are Unicode characters in those paths that are taking more than a single byte to represent? Seems very unlikely/impossible.

The best thing to start with would probably be to check the PATH_MAX and NAME_MAX definitions on your system. If you can't find the header using find, you can probably write a simple C program to print out their values, assuming your compiler is set up to find them.

[xenoscopic]

This could also be a limitation of AFS (if that's what you're using). Filesystems can technically have name length limits that are shorter than the system limit. If you are using AFS, can you try going to the AFS root and doing something like touch efficiencySF.SINGLE_E_2015_e24_lhmedium_L1EM20VH_OR_e60_lhmedium_OR_e120_lhloose_2016_e24_lhtight_nod0_ivarloose_OR_e60_lhmedium_nod0_OR_e140_lhloose_nod0.LooseAndBLayerLLH_d0z0_v11_isolLooseTrackOnly.NEW.root?

[xenoscopic]

I think this is an issue with AFS. Even on Windows, the most limited platform, Go added supported for handling long paths automatically in Go 1.8 (https://golang.org/cl/32451). There was some earlier discussion about this not working with all APIs (golang/go#10651 (comment)), but it seems like that was bogus (golang/go#10577 (comment)).

I'll close for now, but please feel free to re-open if the issue can be reproduced outside AFS. I'd be happy to fix it if I can reproduce.