crash caused by suspected race after last file in a directory is deleted about goofys

[dqh-au]

An ls on the parent directory of the mountpoint showed the following:

# ls -l
ls: cannot access files: Transport endpoint is not connected
total 0
d????????? ? ? ? ?            ? files

[dqh-au]

I found this in /var/log/syslog - timestamp is at the time the mountpoint appeared to fail:

<FULL PATH TO>/goofys[848]: fuse.ERROR Ref 2 report refcnt == 0

[kahing]

seems like goofys has crashed. Do you have a complete log?

[nickschuch]

Also seeing the same thing, will see if I can get a full log as well.

[dqh-au]

It happened again a couple of days ago. How can I get a complete log? That fuse error us the only thing in /var/log/syslog . I'm using up to date Ubuntu 14.04.4 LTS

[kahing]

It's usually easier to log to a file if you manually mount instead of going through fstab. goofys will log to stderr if you pass it -f:

goofys --debug_fuse --debug_s3 -f -o allow_other --uid=1001 --gid=1002 --dir-mode=0750 --file-mode=0640 --region=eu-west-1 --profile=<AWS CONFIG PROFILE> <BUCKET> <MOUNTPOINT> >& goofys.log

Alternatively you can just add --debug_fuse and --debug_s3 to your fstab but will generate A LOT of output to syslog.

[dqh-au]

I've captured a crash log, including a stack trace. Attached file includes output preceding the crash.

goroutine 52356 [running]:
panic(0x867b40, 0xc8203ce0f0)
    /usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/kahing/goofys/internal.(*Inode).Ref(0xc820666240)
    /usr/local/gopath/src/github.com/kahing/goofys/internal/handles.go:77 +0x398
github.com/kahing/goofys/internal.(*Goofys).LookUpInode(0xc82000aa00, 0x7f3831ff58d0, 0xc82066ccc0, 0xc82068de10, 0x0, 0x0)
    /usr/local/gopath/src/github.com/kahing/goofys/internal/goofys.go:557 +0x121
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc82040e220, 0xc8200ba240, 0x7f3831ff58d0, 0xc82066ccc0, 0x83d700, 0xc82068de10)
    /usr/local/gopath/src/github.com/jacobsa/fuse/fuseutil/file_system.go:128 +0x2c9
created by github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps
    /usr/local/gopath/src/github.com/jacobsa/fuse/fuseutil/file_system.go:108 +0x18f

goofys.crash.txt

[dqh-au]

I was running goofys in foreground mode, it had crashed. Running it again failed with the following message until I unmounted the mount point.

-----------------------------------------------------
2016/03/31 11:32:55.235230 main.FATAL Mounting file system: Mount: Statting mount point: stat <<< MOUNT POINT >>>: transport endpoint is not connected

[dqh-au]

Interestingly I had a subcloud (early version of s3fs) mountpoint fail in the same way this morning. Must be something to do with fuse on Ubuntu 14.04 LTS, and some assumption that subcloud, s3fs & goofys are all violating. Or just a bug in that version of fuse.

[kahing]

I suspect the problem is this:

1. suppose there are objects report/1, report/2, report/3, goofys represents them as report/ and files 1, 2, and 3 under report/
2. now you delete the 3 files
3. report/ is no longer on S3
4. goofys thinks that a file was deleted without it knowing and thus some internal assumption is violated

[kahing]

I am having trouble reproducing the crash. Could you describe your workload or provide a test case?

[dqh-au]

I have a bucket mounted and accessed over SFTP. The SFTP client lists the folder containing the incremental report files, then downloads & deletes each file until none remain - so I think you're probably on the track with your theory. The client connects and performs this operation every hour, and the mountpoint usually fails after a few days.

[kahing]

I committed a01c2a3 which should fix your problem. Could you verify that it's fixed since I can't reproduce the problem?

[dqh-au]

Thank you! I'll deploy today and see how it goes. I've left debug flags on so if it happens again i'll have a stack trace for it.

[dqh-au]

Still working nicely a few days in, if it holds for a couple of weeks then i'd say it is fixed.

One extra detail that might help you build a regression test case for this - the files were two 'folders' deep, and nothing else was in the root folder. For example /report/incremental/1.xml, /report/incremental/2.xml , nothing in /report directly. In S3 world deleting 1.xml & 2.xml would result in both /report/incremental & /report no longer being visible because they don't really exist.

[kahing]

I modified the test case in cdae435 to use /report/incremental, reverted the fix, and still couldn't reproduce it :-(

feel free to comment and reopen if this still happens.