jacobsa / fuse Goto Github PK

As you said you would prefer one lump report instead of isolated issues, here is a (non-exhaustive) list. Implementation requires more than just defining constants, hence the different issues pertaining to each. The rationale behind each is also completely different.

The following system error codes are missing as constants in the fuse package

• EACCESS
• EBUSY - for network filesystems
• EAGAIN - for network filesystems
• ENODATA
• EISDIR - for OpenFile on directories
• EPERM
• ERANGE - for directory listings (see: #53 for details)
• EEXDEV
• EBADF - for close(2) and friends (see #55 for details)
• EROFS - for read-only filesystems (see #56 for details)

reproducible with gcsfuse, I always get:

2015/10/16 11:50:59.321987 mount helper error: fusermount: mount failed: Invalid argument

Hey @jacobsa!

Mounting with the fuse package fails with:

callMount: fork/exec /Library/Filesystems/osxfusefs.fs/Support/mount_osxfusefs: no such file or directory

Looks like the osxfuse in 3.5.2 version changed its structure, now the binary is under /Library/Filesystems/osxfuse.fs/Contents/Resources/mount_osxfuse, however I'm not sure it's enough to fix the path.

Thus this issue.

I'm having trouble mounting on demand with autofs, it hangs when trying to mount such as when trying to list the mount folder contents. I'm doing this with gcsfuse google buckets implementation, I have a docker container set up with a test, just add service account details and your bucket name:

$ cat Dockerfile 
FROM levkuznetsov/gcsfuse-docker

ADD service-account.json /etc/gcloud/service-account.json
ADD auto.gcsfuse /etc/autofs/auto.gcsfuse

$ cat auto.gcsfuse 
dfci-mev-def-12345-678	-fstype=gcsfuse,rw,key_file=/etc/gcloud/service-account.json	:dfci-mev-def-12345-678

Where dfci-mev-def-12345-678 is my bucket and the Google service account is in service-account.json. Running in privileged mode and the autofs4 module loaded on the host. Output from gcsfuse is:

attempting to mount entry /mnt/dfci-mev-def-12345-678
>> Calling gcsfuse with arguments: -o rw --key-file /etc/gcloud/service-account.json dfci-mev-def-12345-678 /mnt/dfci-mev-def-12345-678
>> Using mount point: /mnt/dfci-mev-def-12345-678
>> Opening GCS connection...
>> Opening bucket...
>> Mounting file system...

and then it just hangs forever. I believe it's blocking in mount.go#Mount but I have no idea why. I'm able to mount the bucket normally within the container

I could not find an error code matching syscall.ENODATA defined in this package.

It should be defined.

Naked returns make the code harder to read. This is not just my preference, but also documented in the Go CodeReviewComments wiki page.

According to bfleischer in osxfuse/osxfuse#199, osxfuse ignores the entry_valid field in fuse responses. (This maps to EntryExpiration in this package.)

This is a problem for "remote" file systems that want to offer particular consistency guarantees like close-to-open. They might work fine on Linux, but silently fail to be consistent on OS X. My issue here is with the silent failure; I think it should be safe but slow by default, and performant but with relaxed guarantees with opt-in.

So, action items:

• Find some way to write tests for caching, covering both attribute caching (in all relevant methods) and entry caching. Make sure they currently fail on OS X. They presumably pass on Linux.
• Shave the yaks necessary to support arbitrary mount options in the bazil.org package (cf. bazil/fuse#77). Either send a pull request, find some way to hackily mint options using the reflect package, or fork the bazil.org package and add what we want.
• Have the Mount function set novncache when on OS X. If we later want to allow relaxed guarantees but with entry caching, then we can expand the function to take a struct with an "enable vnode caching" option.
• Document that EntryExpiration is always treated as zero on OS X, with call out to the behavior of Mount.
• Make sure the caching tests now pass.

"mount: invalid argument"
this is the strace log
strace.log

@stapelberg thanks

sync.Pool is a per-P (logical processor) data structure (with custom runtime support), which might come with performance wins.

github.com\jacobsa\fuse\internal\fusekernel\fuse_kernel.go:171: undefined: syscall.O_ACCMODE
jacobsa\fuse\internal\fusekernel\fuse_kernel.go:397: undefined: Attr

Hello,

I'm trying to decide on which Fuse Go library to use, either this or https://github.com/bazil/fuse. I got a prototype working using Bazil's Fuse, however after spending most of the day reading your code and comments, I'm inclined to switch over.

One of my requirements is Windows support, was wondering if you were planning to add it eventually (<6 months)? Bazil doesn't support Windows currently either, but I recall reading an issue or comment that it might be supported in the long run.

PS thanks for taking the time to write such detailed comments, it has helped me immensely in understanding the internals of Fuse; especially all the gotchas and warnings.

We define a ForgetInodeOp, but don't currently ever issue it. Start issuing it. Things:

• Give up on the idea of reference counting within the fuse package and just send the count through to the user. They can naturally track counts in their inode structure.
• Be very precise in the documentation about what increments the count.

Debian BTS: https://bugs.debian.org/860703

fails to compile on x86 platform (doesn't occur on x64/amd64), log is as following:

src/github.com/jacobsa/fuse/internal/buffer/runtime.go:22: missing function body for "memclr"
src/github.com/jacobsa/fuse/internal/buffer/runtime.go:27: missing function body for "memmove"

src/github.com/jacobsa/fuse/fusetesting/stat_linux.go:33: cannot use sys.(*syscall.Stat_t).Nlink (type uint32) as type uint64 in assignment

the issue of last line is resolved by my patch: https://bugs.debian.org/860703#10
but the previous 2 lines still occur. please kindly help. thank you!

"golang.org/x/net/context"

this has been builtin package in 1.7.

I was trying to use this library https://github.com/kahing/goofys and getting an error from fuse:

panic: Page size is unexpectedly 65536

goroutine 1 [running]:
github.com/jacobsa/fuse/internal/buffer.init.0()
	/root/gopath/src/github.com/jacobsa/fuse/internal/buffer/in_message.go:33 +0xe4

So I assume it has to do with fuse.
Any hint on what it should change in order to get it working for ppc64le architecture?

In order to figure out how the interface methods relate to various command line tools (stat, ls, touch, echo for all the various file manipulation), I tried to use the following code to return custom errors... Instead I am getting this error for almost all operations I can try: ls: cannot access 'mnt': Input/output error

note: pay attention to some of the comments as they explain the different ways I've tried to use this library.

package main

import (
	"context"
	"errors"

	"github.com/dgraph-io/badger"
	"github.com/jacobsa/fuse"
	"github.com/jacobsa/fuse/fuseops"
	"github.com/jacobsa/fuse/fuseutil"
)

// NOTE:
// requires osxfuse to be installed for MacOS

func check(err error) {
	if err != nil {
		panic(err)
	}
}

type DbFs struct {
	// tried with and without this type
	// fuseutil.NotImplementedFileSystem
}

func (d *DbFs) StatFS(ctx context.Context, op *fuseops.StatFSOp) error {
	return errors.New("unable to perform stat, stat not yet impl")
}

func (d *DbFs) LookUpInode(ctx context.Context, op *fuseops.LookUpInodeOp) error {
	return errors.New("unable to perform lookup, lookup not yet impl")
}

// This method seems to cause the error, if this doesn't exist
// "function not implemented" is the error
func (d *DbFs) GetInodeAttributes(ctx context.Context, op *fuseops.GetInodeAttributesOp) error {
	// tried an failed
	// op.AttributesExpiration = time.Now().Add(time.Hour * 10)
	// op.Attributes = fuseops.InodeAttributes{
	// 	Nlink: 1,
	// 	Mode:  0777,
	// 	Size:  0,
	// }
	return errors.New("unable to perform getinode, getinode not yet impl")
}

func (d *DbFs) SetInodeAttributes(ctx context.Context, op *fuseops.SetInodeAttributesOp) error {
	return errors.New("unable to perform setInode, setInode not yet impl")
}

func (d *DbFs) ForgetInode(ctx context.Context, op *fuseops.ForgetInodeOp) error {
	return errors.New("unable to perform mkdir, mkdir not yet impl")
}

func (d *DbFs) MkDir(ctx context.Context, op *fuseops.MkDirOp) error {
	return errors.New("unable to perform mkdir, mkdir not yet impl")
}

func (d *DbFs) MkNode(ctx context.Context, op *fuseops.MkNodeOp) error {
	return errors.New("unable to perform mknode, mknode not yet impl")
}

func (d *DbFs) CreateFile(ctx context.Context, op *fuseops.CreateFileOp) error {
	return errors.New("unable to perform createfile, createfile not yet impl")
}

func (d *DbFs) CreateSymlink(ctx context.Context, op *fuseops.CreateSymlinkOp) error {
	return errors.New("unable to perform createsymlink, createsymlink not yet impl")
}

func (d *DbFs) Rename(ctx context.Context, op *fuseops.RenameOp) error {
	return errors.New("unable to perform rename, rename not yet impl")
}

func (d *DbFs) RmDir(ctx context.Context, op *fuseops.RmDirOp) error {
	return errors.New("unable to perform rmDir, rmDir not yet impl")
}

func (d *DbFs) Unlink(ctx context.Context, op *fuseops.UnlinkOp) error {
	return errors.New("unable to perform unlink, unlink not yet impl")
}

func (d *DbFs) OpenDir(ctx context.Context, op *fuseops.OpenDirOp) error {
	return errors.New("unable to perform opendir, open not yet impl")
}

func (d *DbFs) ReadDir(ctx context.Context, op *fuseops.ReadDirOp) error {
	return errors.New("unable to perform readdir, readdir not yet impl")
}

func (d *DbFs) ReleaseDirHandle(ctx context.Context, op *fuseops.ReleaseDirHandleOp) error {
	return errors.New("unable to perform releasedirhandle, releasedirhandle not yet impl")
}

func (d *DbFs) OpenFile(ctx context.Context, op *fuseops.OpenFileOp) error {
	return errors.New("unable to perform openfile, openfile not yet impl")
}

func (d *DbFs) ReadFile(ctx context.Context, op *fuseops.ReadFileOp) error {
	return errors.New("unable to perform readfile, readfile not yet impl")
}

func (d *DbFs) WriteFile(ctx context.Context, op *fuseops.WriteFileOp) error {
	return errors.New("unable to perform writefile, writefile not yet impl")
}

func (d *DbFs) SyncFile(ctx context.Context, op *fuseops.SyncFileOp) error {
	return errors.New("unable to perform getxattr, getxattr not yet impl")
}

func (d *DbFs) FlushFile(ctx context.Context, op *fuseops.FlushFileOp) error {
	return errors.New("unable to perform getxattr, getxattr not yet impl")
}

func (d *DbFs) ReleaseFileHandle(ctx context.Context, op *fuseops.ReleaseFileHandleOp) error {
	return errors.New("unable to perform getxattr, getxattr not yet impl")
}

func (d *DbFs) ReadSymlink(ctx context.Context, op *fuseops.ReadSymlinkOp) error {
	return errors.New("unable to perform getxattr, getxattr not yet impl")
}

func (d *DbFs) RemoveXattr(ctx context.Context, op *fuseops.RemoveXattrOp) error {
	return errors.New("unable to perform getxattr, getxattr not yet impl")
}

func (d *DbFs) GetXattr(ctx context.Context, op *fuseops.GetXattrOp) error {
	return errors.New("unable to perform getxattr, getxattr not yet impl")
}

func (d *DbFs) ListXattr(ctx context.Context, op *fuseops.ListXattrOp) error {
	return errors.New("unable to perform listxattr, listxattr not yet impl")
}

func (d *DbFs) SetXattr(ctx context.Context, op *fuseops.SetXattrOp) error {
	return errors.New("unable to perform setxattr, setxattr not yet impl")
}

func (d *DbFs) Destroy() {}

func main() {
	dstore := "./atfs"
	opts := badger.DefaultOptions

	opts.Dir = dstore
	opts.ValueDir = dstore

	db, err := badger.Open(opts)
	defer db.Close()
	check(err)

	fs := &DbFs{}
	server := fuseutil.NewFileSystemServer(fs)

	mfs, err := fuse.Mount("./mnt", server, &fuse.MountConfig{})
	check(err)

	check(mfs.Join(context.Background()))
}

There was recently a bunch of work on bazil.org/fuse, to support a new fuse protocol. At this point the mirror jacobsa/bazilfuse, on which this package relies, is significantly out of date. Update it.

Commits that potentially need attention:

• bazil/fuse@3ad4898: Init is no longer exposed. Will need to look at what options gcsfuse sets here, and consider sending a pull request to use them or make it configurable. Maybe the mount options added in bazil/fuse@4ae50a4 and bazil/fuse@68fc4be are what we need?

Don't forget to run all fuse and gcsfuse tests on OS X and Linux.

I saw that bazil.org/fuse appeared in this project's readme.
Can you give some description how this project compared to bazil.org/fuse?

Thanks

Are there any plans to support the sync filesystem operation? I need to to force the filesystem to upload all dirty files that it has. The unmount operation doesn't wait for this to happen, unfortunately.

I could not find an error code matching syscall.EPERM defined in this package.

It should be defined.

This package isn't building on my system.

I am using the following command: go get -u github.com/jacobsa/fuse

I am getting the following errors:
../../go/src/github.com/jacobsa/fuse/internal/buffer/runtime.go:22: missing function body for "memclr"
../../go/src/github.com/jacobsa/fuse/internal/buffer/runtime.go:27: missing function body for "memmove"

Looking at these files hints at the fact that those are supposed to be entry points for their C counterparts, but I don't know Go well enough to figure out what's really going on.

I am running Arch Linux ARM on a Raspberry Pi 2.

Thanks for sharing this wonderful library in Go.

I'd like to query if any one is working on implementating InvalidateNode/InvalidateEntry here? I see #61 basically asked the same but stay open atm.

This has been implemented in bazil.org/fuse (which I suppose is the kernel part's upstream):
https://github.com/bazil/fuse/blob/65cc252bf6691cb3c7014bcb2c8dc29de91e3a7e/fuse.go#L1156

I think porting that implementation here and make all tests pass should be enough. If no other one is working on it, I'd like to draft a PR for review, but your directions & suggestions should be helpful before I start the work.

Background info is, I'm writing a homegrown array database, that based on ZFS files shared over NFS to many computing nodes. But close-to-open cache consistency of NFS is becoming a problem, and I'd take the chance to map may small array data files into few large virtual data files to be mmap'ed, so inode invalidation is a tight requirement. And since ZFS already been the inode/unix thing, the approach at bazil that to manage virtual inodes on the fly feels inappropriate.

Currently all the fuse operations are handled in parallel. C-fuse has a sync_read option which performs all the reads synchronously, which is useful when the filesystem wants to perform its own readahead, for example.

Ideally this would be per fd.

Hi,

I met following problem when I use goofys on fuse.

It seems LookUpInode for file "record-00063-of-00100" happened twice as same time, then ForgetInode called twices, which lead to the panic.

I checked source code at the call stack, but didn't find any clue about this problem.
I don't have much knowledge about this. I have some questiongs.
How or Why ForgetInode is trigger? And what is ForgetInode op for?

Could someone help me about this? Thanks in advance!

2019/01/04 19:55:46.420179 fuse.DEBUG Op 0x001732af        connection.go:395] <- LookUpInode (parent 4, name "record-00063-of-00100")
2019/01/04 19:55:46.420305 fuse.DEBUG Inode.LookUp 4 record/files [record-00063-of-00100]
2019/01/04 19:55:46.420652 fuse.DEBUG Op 0x001732b0        connection.go:395] <- LookUpInode (parent 4, name "record-00063-of-00100")

...
2019/01/04 19:55:46.420778 fuse.DEBUG Inode.LookUp 4 record/files [record-00063-of-00100]
...
2019/01/04 19:55:46.422393 fuse.DEBUG <-- LookUpInode 4 record-00063-of-00100 <nil>
...
2019/01/04 19:55:46.422445 fuse.DEBUG <-- LookUpInode 4 record-00063-of-00100 <nil>
...
2019/01/04 19:55:46.422453 fuse.DEBUG Op 0x001732af        connection.go:478] -> OK (inode 12242)
...

2019/01/04 19:55:46.422504 fuse.DEBUG Op 0x001732b0        connection.go:478] -> OK (inode 12242)
2019/01/04 19:55:46.422556 fuse.DEBUG Op 0x001732b2        connection.go:395] <- ForgetInode (inode 12242)
2019/01/04 19:55:46.422577 fuse.DEBUG DeRef 12242 record/files/record-00063-of-00100 [1 1]
2019/01/04 19:55:46.422774 fuse.DEBUG Op 0x001732b2        connection.go:478] -> OK ()
2019/01/04 19:55:46.422797 fuse.DEBUG Op 0x001732b3        connection.go:395] <- ForgetInode (inode 12242)
panic: Unknown inode: 12242

goroutine 44 [running]:
github.com/kahing/goofys/internal.(*Goofys).getInodeOrDie(0xc42021dd40, 0x2fd2, 0xc4e90c4cc0)
        /opt/bin/work/src/github.com/kahing/goofys/internal/goofys.go:421 +0x126
github.com/kahing/goofys/internal.(*Goofys).ForgetInode(0xc42021dd40, 0xa01b00, 0xc599933650, 0xc4257327d0, 0x899000, 0xc4257327d0)
        /opt/bin/work/src/github.com/kahing/goofys/internal/goofys.go:696 +0x58
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc42036a2e0, 0xc420388c30, 0xa01b00, 0xc599933650, 0x899000, 0xc4257327d0)
        /opt/bin/work/src/github.com/jacobsa/fuse/fuseutil/file_system.go:151 +0x8c4
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps(0xc42036a2e0, 0xc420388c30)
        /opt/bin/work/src/github.com/jacobsa/fuse/fuseutil/file_system.go:119 +0x11c
github.com/jacobsa/fuse.Mount.func1(0x9fcaa0, 0xc42036a2e0, 0xc420388c30, 0xc420384070)
        /opt/bin/work/src/github.com/jacobsa/fuse/mount.go:89 +0x3f
created by github.com/jacobsa/fuse.Mount
        /opt/bin/work/src/github.com/jacobsa/fuse/mount.go:88 +0x497

I'm guessing I am doing something silly, but all the ctime/mtime values as shown by the operating system are way off, like from the year 1754 off. In looking at the sample code, time.Now() is used quite often, which is what I'm using so I'm at a loss as to what silly thing I'm doing wrong. Might you have any hints for me?

It seems in the OSX, the ReadFileOp.Dst callback always set Dst with 8K size, is there anyway to bump this size to a bigger number?

As per the documentation on open(2), they can return EISDIR as a more specific error code when attempting to use this method on a directory.

I could not find an error code matching syscall.EISDIR defined in this package.

You defined ENOTDIR but not this code...

It should be defined.

Do you plan to add xattr support?

I could not find an error code matching syscall.EAGAIN defined in this package.

It should be defined.

If I work around #3 by removing the go keyword and processing requests serially, I often get a deadlock with:

go test -c ./samples/flushfs  -o /tmp/foobar && (for i in {0..1000}; do /tmp/foobar --fuse.debug --ogletest.run FlushFSTest.Mmap_CloseBeforeMunmap|| exit 1; done) && echo OKAY

The kernel state looks like this:

jacobsa@fourier:/sys/fs/fuse/connections% pstree -p 14829                                                                                                               [1/130]
foobar(14829)─┬─{foobar}(14830)
              ├─{foobar}(14831)
              └─{foobar}(14833)


jacobsa@fourier:/sys/fs/fuse/connections% cat /proc/14829/stack
[<0000000000000000>] wait_answer_interruptible+0x6a/0xa0
[<0000000000000000>] __fuse_request_send+0x1fb/0x280
[<0000000000000000>] fuse_request_send+0x12/0x20
[<0000000000000000>] fuse_readpage+0x152/0x1e0
[<0000000000000000>] filemap_fault+0x116/0x410
[<0000000000000000>] __do_fault+0x6f/0x530
[<0000000000000000>] handle_mm_fault+0x482/0xf00
[<0000000000000000>] __do_page_fault+0x184/0x560
[<0000000000000000>] do_page_fault+0x1a/0x70
[<0000000000000000>] page_fault+0x28/0x30
[<0000000000000000>] 0xffffffffffffffff


jacobsa@fourier:/sys/fs/fuse/connections% cat /proc/14830/stack
[<0000000000000000>] call_rwsem_down_write_failed+0x13/0x20
[<0000000000000000>] vm_mmap_pgoff+0x6c/0xc0
[<0000000000000000>] SyS_mmap_pgoff+0x116/0x270
[<0000000000000000>] SyS_mmap+0x22/0x30
[<0000000000000000>] system_call_fastpath+0x1a/0x1f
[<0000000000000000>] 0xffffffffffffffff


jacobsa@fourier:/sys/fs/fuse/connections% cat /proc/14831/stack
[<0000000000000000>] futex_wait_queue_me+0xdd/0x140
[<0000000000000000>] futex_wait+0x182/0x290
[<0000000000000000>] do_futex+0xde/0x760
[<0000000000000000>] SyS_futex+0x71/0x150
[<0000000000000000>] system_call_fastpath+0x1a/0x1f
[<0000000000000000>] 0xffffffffffffffff


jacobsa@fourier:/sys/fs/fuse/connections% cat /proc/14833/stack
[<0000000000000000>] ep_poll+0x262/0x340
[<0000000000000000>] SyS_epoll_wait+0xd5/0x100
[<0000000000000000>] system_call_fastpath+0x1a/0x1f
[<0000000000000000>] 0xffffffffffffffff

It looks like what is happening here is that the test goroutine is page faulting and holding mm_struct::mmap_sem while another running goroutine is attempting to mmap (blocking on the down_write call in vm_mmap_pgoff). Because we see a deadlock, presumably this is the goroutine that is serving fuse read requests (perhaps it is mmap'ing on behalf of a buffer allocation).

Unlike 2728137, I don't see a way to work around this with a single daemon/test process. Argh. Instead, I think we'll need to run the file system in a separate process. At that point, I think we can remove the GOMAXPROCS workaround from before.

I'm trying to build a different project (kahing/goofys), which uses this project. I'm using a Rock64 running Ubuntu 17.10 armhf, actually aarch64 architecture. I get this error message:

It's not clear to me what happens if we return an error here. Find out in the kernel, and add a note to the documentation.

As per the documentation on any write type action, this error code should be returned when attempts are made against a filesystem intended to be read-only (either through use of a mount option or by design.)

I could not find an error code matching syscall.EROFS defined in this package.

It should be defined.

This can be used as a more specific error code to an extended (streamed) directory listing, and to ReadDir when the requested offset is beyond the range of the parent directory's file listing.

I could not find an error code matching syscall.ERANGE defined in this package.

It should be defined.

Hello up there.

I'm currently using go-fuse library for my filesystem, but from time to time I'm also rechecking on how it goes with jacobsa/fuse. In 2018 jacobsa/fuse was ~1.6x slower (2) compared to hanwen/go-fuse, and today (2020, Mar.11) I've found that hanwen/go-fuse improved its throughput, while jacobsa/fuse stays at approximately the same performance level(*). Today, the ratio in speed is ~3x (5.7 GB/s vs ~ 1.9 GB/s). edit: today the ratio is ~1.8x - see #78 (comment).

Is there anything I'm missing?

I've looked only briefly and could indeed miss something obvious which could be related to performance or generally. In such a case I appologize for creating the noise.

Thanks beforehand for feedback,
Kirill

/cc @jacobsa, @stapelberg

(*) both at current master and even if I try to patch it to handle requests directly instead of in a goroutine (1, 2).

As per the documentation on close(2) and close(3), when a file handle either does not exist or has gone stale, or has already been closed; this error should be returned. Currently, one must return EIO or EINVAL as you have not given us the proper options.

I could not find an error code matching syscall.EBADF defined in this package.

It should be defined.

I could not find an error code matching syscall.EBUSY defined in this package.

It should be defined.

I could not find an error code matching syscall.EEXDEV defined in this package.

It should be defined.

UPDATE, 2015-04-02: Make sure to see also issue #8. Much of this is incorrectly diagnosed, due to a kernel bug.

If I run the test FlushFSTest.Mmap_MunmapBeforeClose 1,000 times waiting for the first error, like so:

go test -c ./samples/flushfs  -o /tmp/foobar && (for i in {0..1000}; do /tmp/foobar --ogletest.run FlushFSTest.Mmap_MunmapBeforeClose --fuse.debug || exit 1; done) && echo OKAY

it almost always fails with something like the following:

flush_fs_test.go:627:
Expected: elements are: [paco]
Actual:   [taco], whose element 0 doesn't match

Running with --fuse.debug shows this is because the second write request (from the modification to the mapped memory) is received after the flush for the close(2), which happens before the modification in test sequence.

This is puzzling, given the code walk documented on WriteFile which appears to prove that fuse writes out all dirty pages before sending a flush request:

• (http://goo.gl/PheZjf) fuse_flush calls write_inode_now, which appears to start a writeback in the background (it talks about a "flusher thread").
• (http://goo.gl/1IiepM) fuse_flush then calls fuse_sync_writes, which "[waits] for all pending writepages on the inode to finish".
• (http://goo.gl/zzvxWv) Only then does fuse_flush finally send the flush request.

The comments there talk about pending writepages requests, but don't use the term "dirty". The commit that introduced this behavior, torvalds/linux@fe38d7d is more clear it talks directly about "dirty pages".

I modified the way requests are logged by server.go, and I believe the issue here is simply that the goroutines spawned by server.Serve are racing. That is, fuse is sending "write, write, flush", but we are processing them as "write, flush, write" because fuse sends write requests but doesn't wait for them.

This is caused by bazil/fuse#66. Plan:

• Uncomment workarounds for this in fuse tests (for example here) so that they are failing.
• Fix the broken behavior in jacobsa/bazilfuse. Do it in a branch off of the upstream branch.
• Make sure the tests pass.
• Re-enable the corresponding assertions in jacobsa/gcsfuse, too.

I could not find an error code matching syscall.EACCESS defined in this package.

It should be defined.

How does this library compare to the other two Golang FUSE libraries? bazil/fuse docs are horrible and completely in flux, go-fuse seems a little more level than I need... How does this library compare in terms of performance and api?

Implement fuse_lowlevel_notify_inval_inode. This should allow user space processes to trigger eviction of the kernel's inode cache, and in turn getting the "ForgetInodeOp" to evict its own data.

Context: https://sourceforge.net/p/fuse/mailman/message/30057855/

It seems like libfuse initializes the dir entry's inode to this value (-1), rather than zero.

Because of what appears to be a kernel bug, it looks like I was totally confused about the "race" in issue #3. In particular:

• The kernel guarantees that it will handle request serialization when the user expects it:

AFAIU, kernel fuse never sends next request before getting ACK for
previous one if user expects corresponding operations to be ordered.

• The fact that the "no msync, munmap before close" test fails when the write sleeps in the background and the flush is allowed to proceed appears to be a bug that was fixed somewhere between 3.13 and 3.16.

To do:

• Put some forensics in this thread about the bug, etc. Failure output and patches that can reproduce it. (Currently it can be reproduced at a191868279618d6882fe2f3281565bd956b14745, but that may be garbage collected.)
• Relax the stern warnings in the documentation about request serialization. Both in package fuse and package fuseops. Grep "FlushFileOp", "SyncFileOp", "parallel", "serial", "msyc". Do note the kernel bug, though.
• When updating the docs, do make sure to still pay attention to what OS X does.
• Consider making fileSystemServer call each file system method on its own goroutine. Also consider adding a flag-controlled random sleep at the start of the goroutine, to shake out further bugs.

Hi, would be great a sample about mounting a zip file as described here

in the example are using bazil, I found a bit confusing the documentation, so I'd appreciate if you can give me a similar example using your code, so I could understand the differences between both projects

thank you so much!!!

For the sake of encapsulation, make sure that bazilfuse doesn't show up anywhere in the public interface. This gives us the flexibility to switch back and forth between forks more easily.

Hi,

I'm trying to enable FTP/SFTP access to Google Cloud bucket, by using Pure-FTP server on Google instance. However I'm experiencing some weird issue when I'm trying to delete file trough FTP, you can see this issue in more details here.

I'm suspecting that the ftp server is failing to address the DELETE request properly due to some small delays in gcsfuse mounted partitions.

So my question is if it's there any way that I can solve this by tuning up gcsfuse performance. My current mount settings are:

backup /home/backup gcsfuse rw,auto,allow_other,uid=1001,gid=1001,file_mode=0777,dir_mode=0777

Thanks

I am shaving yaks for GoogleCloudPlatform/gcsfuse#68:

• I want the gcsfuse file system to spawn a garbage collector process for temporary blobs when it is mounted, and kill it off when the file system is unmounted (by cancelling its context, etc).
• In order to do this, I need the file system to have some way to be told when
  it will no longer receive any operations because it is being unmounted. But on Linux, the kernel doesn't reliably send destroy messages.
• So instead I need to regard fuse.Connection.ReadOp returning io.EOF as a "destroy" message. Therefore fuseutil.fileSystemServer.ServeOps needs to call a FileSystem.Destroy method when ServeOps returns.
• But I want to guarantee that a call to FileSystem.Destroy doesn't race with calls to other file system methods, which are done in other goroutines. So I need to wait for all in-flights to finish first.
• Currently that waiting is done in fuse.Connection.close, which is the wrong place; it belongs where the concurrency is introduced (i.e. fileSystemServer). So I need to move the sync.WaitGroup to there.
• But fileSystemServer has no way to see when the file system is done with an op, because the file system may call Op.Respond on another goroutine; it doesn't guarantee to block until it has responded.

Wow. So:

• Move op responding into fileSystemServer, updating the FileSystem interface to return the error that should be used for responding.
• Move the sync.WaitGroup into fileSystemServer, out of fuse.Connection. Don't forget to update comments on fuse.Connection.Close.
• Add a FileSystem.Destroy method that guarantees to happen after all other methods return, with no future calls to any method. Update samples/forgetfs to use this to zero inode lookup counts, and remove the workaround from the Check method on Linux.

By using := in the following places, I suppose the err variable declared in the return list is shadowed:

Isn't this a bug or I missed something?