xoofx / zio Goto Github PK

The documentation for IFileSystem.EnumeratePaths mentions a remarks section, but no remarks section exists.

Hello,

I was doing some integration tests to see if I could incorporate Zio in some of my projects, and while Testing, I could see that UPath(null) is allowed.

I was curious if there was any benefit for it (versus throwing a constructor exception) :

• In most cases calling any filesystem method will result into throwing some exceptions
• Calling combine will also throw exceptions.
• In some cases those are just replaced silently by an empty path, but maybe just providing empty path in the first place would make more sense.

So allowing initial null input feels like a little bit like an exception just waiting to be thrown deeper in the software, and disallowing null path early would look like to help preventing.

Another short one (to stay in the topic), UPath to string implicits/explicits are reversed (in .net conventions).

UPath constructor can throw exception, so string -> UPath operator should be explicit, not implicit (implicit should guarantee that no exception not information loss will happen)

on the other hand, UPath to string could be implicit, as every UPath can be a string.

As a side note, I'm quite happy to PR those changes (eg : removing IsNull, and adding initial guard), but wanted to discuss them beforehand, just in case I missed something
Thanks

I'm trying to adopt a VFS for testing and my app uses symlinks. I didn't see support in the IFileSystem API. Curious whether that's desired.

In theory it could be part of a new ISymlinkFileSystem interface instead. That would presumably prevent things like ZipFileSystem from having to implement support (or throw, more likely).

Right now using any of the ComposeFileSystems will lead to missed Dispose calls because none of them call Dispose on the filesystems they have references to.

IMO they should (optionally) take ownership of the filesystem instances and dispose of them as necessary. Either when they get disposed or they are being removed from the composed filesystem.

Just hit this:

var fs = new SubFileSystem(new PhysicalFileSystem(), realPath));

I assume the proper way to do this is:

var physicalFs = new PhysicalFileSystem();
var fs = new SubFileSystem(physicalFs, physicalFs.ConvertPathFromInternal(realPath)));

If there weren't an implicit conversion, I probably would have guessed that. Overall, it seems like keeping UPaths and strings a bit separate is a good idea. Especially when working with subpaths like this, the chances that you'll confuse one for the other is pretty high.

Six tests fail on my machine at this line:

other.AllFiles_txt does not contain any of the files with an extension beginning with "txt". Perhaps this behavior was removed from Windows?

Windows 10 version 1709, build 16299.98

Thoughts on cross targeting to .NET Standard 2.0 to simplify the dependencies for modern runtimes (Mono, NETCOREAPP 2.0, NETFramework 4.7.1, etc)?

Trying to avoid taking in all these forwarding libraries (and possible future versioning conflicts).

Just a note, on a couple of things I've come across while working with zio.

I'm using a configuration with a mount file system as the root, which is mounting both physical and memory file systems.

1. I've noticed that using a watcher on the root, produces a lot of change events from the memory file system. I haven't looked into the details, but I guess the implementation is built around a block write or similar. Anyway, if this behavior is not expected, I could take a deeper look to see if I can find something.

2. I've also experienced deadlocks when accessing the MemoryFileSystem. I wonder if the library itself is supposed to be thread safe or not. The file system itself obviously have locks, but I'm not sure if I need to protect thread access to the zio API on top of that.

I was trying to get the pysical path of a file within a mounted file system, but it's not super happy about it,
I assumed the ConvertPathToInternal would return this, but it returns the input path. I then checked the code and noticed it didn't use the mounts at all.

So I cooked this little extension up - maybe someone can improve and add it to a future release.

public static class ZioExtensions
{
    public static string GetPhysicalPath(this IFileSystem fs, UPath path)
    {
        if (fs is MemoryFileSystem) throw new Exception($"Path '{path}' does not have a physical path on MemoryFileSystems");
        if (fs is MountFileSystem mfs) return GetPhysicalPath(mfs, path);
        return fs.ConvertPathToInternal(path);
    }

    public static string GetPhysicalPath(this MountFileSystem mfs, UPath path) {
        if (mfs.TryGetMount(path, out var name, out var fs, out var fsPath)) {
            return fs?.GetPhysicalPath(fsPath ?? throw new Exception($"Path '{path}' does not have a physical path - internal error"));
        }
        try
        {
            return mfs.ConvertPathToInternal(path);
        }
        catch (InvalidOperationException)
        {
            throw new Exception($"Path '{path}' does not have a physical path on this MountFileSystem and no delegate fallback was provided");
        }
    }
}

PS. Sorry for not submitting a PR with tests and everything, I have too much going at the moment...

like:
squashfs
registryfs
ipfs

Calling Dispose should never throw, even if the object is already disposed.

If other file systems are used it seems a bit artificial to have to reference System.IO for the SearchOption enum. Is there a reason for this deign?

Would it be the correct approach to use MemoryFileSystem if I want to create a virtual file system that is persisted to disk in a single file?

At some point I would then have to take the data from the MemoryFileSystem and dump it to disc, is that possible or is there a better way to achieve this?

I would like to read and write on a folder placed on a shared storage using the SubFileSystem. Is this possible? I can't seem to find the way, I always get the exception:
A path on Windows must start by /mnt/ followed by the drive letter.
Also, I'd like to be compatible with both Windows and Linux.

This is the code I have:

UPath root = "\\\\myserver\\PublicFolder\\";
var fs = new PhysicalFileSystem();
if (!fs.DirectoryExists(root))
{
	fs.CreateDirectory(root);
}
var subfs = new SubFileSystem(fs, root);

Thanks

Hi @xoofx,
do you already have any plans for the Git FileSystem?
I am currently working on a project which require this capability. We already using ZIO and the project will benefit from this feature.

Perhapse we can collaborate to improve ZIO with a Git FileSystem.

Thank you for your awesome library!

Unless DirectoryEntries have been created using Zio, DirectoryExists on ZipArchiveFileSystems will return false for any path except the root, despite files containing the path existing.

While zip files themselves have no notion of directories, it is inconsistent and inconvenient if this is not handled uniformly by this abstration layer.

This is occuring while using Zio as a NuGet package on .net 4.6.2

Currently MountFileSystem requires that it either own all mounted file systems and the backup one or none of them. I have a case where I want it to own the backup one but not the mounts. I think it should also be possible to own or not own each mount.

Hi,

First of all, this library is really nice, and well written - very good job :-)

There's a small glitch in that you can actually mount windows file systems with uppercase drive letters and it seems to work fine, until you try and enumerate the directories.

Maybe, it can be more forgiving by aligning the drive letter casing somewhere?

Like the standard System.IO API to create a directory, it processes all segments in the path. However when using something like fs.CreateDirectory("/a/b/c"); in Zio it will fail with System.UnauthorizedAccessException : Cannot create root directory '/'.

I would have expected it to ignore / since it already exists.

Is it possible to run files within the virtual file system using Process.Start?

I have used the ZipArchiveFileSystem to create a zip file on disk. And then I add some directory and files. But how can I save these modifications to disk?

I use the code below:

using Zio;
using Zio.FileSystems;

namespace ZioTest;

internal class ZipCreator
{
    public void CreateZip()
    {
        var path = Environment.CurrentDirectory;
        //Console.WriteLine(path);

        var zipFs = new ZipArchiveFileSystem($"{path}/test1.zip");
        zipFs.CreateDirectory("/test");
        zipFs.WriteAllText("/test/test.txt", "Hello World");
    }
}

public class Program
{
    public static void Main(string[] args)
    {
        ZipCreator creator = new();
        creator.CreateZip();
    }
}

After I run the code, I got the zip file "test1.zip", but it's size is zero.

I have read the document and not found any information about save a FileSystem to disk.

dotnet version: 7.0
zio version: 0.17.0

Hi there,

I think it will be really useful if we have two more functions when we use MemoryFileSystem mode :

1. Load data from physical file into memory when App initialize.
2. Save data from memory into physical file when App exit.

It's a well balance between performance and data value.

Thanks,
Jack

If I have a zip file i can use this to open it into memory without having to extract the whole thing or even a file from it to disk...so I could pass a file that is in the zip to something else that would read a file as if it existed on disk but instead it is in memory?

That right?

Does it work on Android/IOS? I see cross platform just wondering as my intention is to use it with Unity C# where cross platform support is required at least for my use case.

Hello,
I wonder if the library can support to mount from memory to a folder?
Something like:
var fs = new MemoryFileSystem();
fs.CreateDirectory("/Test");
var mountfs = new MountFileSystem();
mountfs.Mount("/mnt/c/Temp", fs);
After that, we can use Explore to browse c:\Temp and see Test folder inside.
Thank you.

Sincerely,
Quang, Vu

The ValidatePath function

• validates that none of the chars is a control character
• then forwards to ValidatePathImpl, which
  • validates that no ':' exists.

AFAIK the only not allowed char in a linux filesystem is a '\0' char. Windows has a few more.

Therefore I'm wondering whether the validation could change to either:

• Only validate the common set => no '\0' exists
  OR
• Allow FS to define their own set of illegal chars to validate via the ctor

I need this functionality and would like to try implementing it. Did you have any additional plans for this @xoofx?

I haven't totally figured out how this works but one of our users managed to escape the restraints of a subfilesystem to be able to write anywhere on the physical filesystem.

The basic theory is this..

using System.IO;
using Zio;
using Zio.FileSystems;

var physical = new PhysicalFileSystem();

// create a sandboxed subsystem
var subsystem = new Zio.FileSystems.SubFileSystem(physical, "/mnt/c/temp/sandboxed", false);

// cool! this is allowed!
subsystem.WriteAllText("/hello.txt", "hello");

// agh!
UPath path = "/\0\0/mnt/windows/system32/evil.txt";
subsystem.WriteAllText(path, "we escaped!");

This seems to want to write to a T: drive (which I don't have).

But obviously there's some deeper funny business going on here.

I'm not sure whether you even meant for the subfilesystems to work in the sandboxed fashion we're using them, but since this seems like unpredictable/unwanted behaviour I thought I would report it.

Because a mount can now exist within another mount. For example we can mount /a and /a/b and if we watch /a it needs to watch both of them, not just /a.

Supporting a virtualized file system in a sandboxed code environment, I would like to be able to (at initialize) support Case sensitive vs. Case Insensitive, it looks like this isn't really an easy configuration change on Zio unless I am missing something. What's the best route?

If more than one filesystem has the same entry at a path the aggregate will list both. I don't consider this desirable because there would be no way to access the duplicate entries except through the original filesystem. Although it's possible and I'm sure someone will have a strange use for it this behavior is inconsistent with how normal file operations work on it.

The filesystem watcher is probably affected as well because it would need to check if a filesystem with higher priority also contains an entry for the path referenced by the event.

Currently this library cannot be consumed from strong-named assembly. Would you accept a PR to add new generated snk file and directives to sign the assembly?

Derivation from a suggestion from @Porges on twitter

Original idea was to extract read methods from IFileStream to a IReadOnlyFileSystem and derive IFileSystem from it... but I'm not sure I like the idea that IFileStream is inheriting from IReadOnlyFileSystem as it would imply that casting to this interface would guarantee that the underlying filesystem is readonly... which maybe not...

So instead, scratching the idea of:

• Add a IReadFileSystem (or IReadableFileSystem?) that contains only read methods from IFileSystem
• Add an abstract class ReadFileSystem that provides default implem of FileSystem
• Make IFileSystem inheriting from IReadFileSystem and FileSystem abstract inheriting from ReadFileSystem

Thoughts?

Hey @xoofx! When you wrote you'd like to "Add support for Git FileSystem (readonly)", did you mean mounting a remote without fetching all files? I looked into this a bit and it seems that's not very straightforward with Git. For one there is no way to get the directory tree from a remote without fetching the actual files. It's also not possible to fetch specific files from a remote. So I don't think a VFS for Git that lazily fetches files can be implemented. At least not for Git directly.

Microsoft recently open-sourced Git Virtual FileSystem (GVFS) which relies on additional operations - specified by their GVFS protocol - for its implementation. If the big Git services were to implement that protocol, implementing a VFS based on that is probably the best course of action.

Of course a VFS could also be implemented specifically for GitHub or any other service with a public API that supports the required operations. Short term, this is what I'd go for (I'm only interested in GitHub at this point).

/a | fs2
file1.txt | fs2
file2.txt | fs1
file4.txt | fs1
/b | fs2
file5.txt | fs2
file3.txt | fs1

Just a typo in the documentation I think - I would expect file4,txt to exist on fs2?

What should a filesystem that doesn't support watching do? I'm thinking returning null might be better than throwing because we should allow watching on composites (aggregate, mount) even when one of the filesystems cannot be watched.

What do you think?

For example if you mount at /x/y/z and then list / it will show /z exists but it should show /x, /x/y, and /x/y/z.

Hello author, first of all this library looks very powerful.
But for the virtual file system, it seems that except for the memory file system, others are not real virtual file systems.
For example: If I want to treat a whole block of large files as a hard disk, a partition to be precise, create folders and files in it, and save the tree structure of the virtual file system. Does this library seem powerless? Yes. But I believe that the essentially similar ones should be the same as the memory file system. They may have almost universal logic except for the storage source.

Of course I'm not sure if my understanding of virtual filesystems is correct.

In TryGetPath we just return here

But we should really only be returning if the path it returns is valid, since it isn't giving the other systems a chance to check for the file.

UPath.Root / ""
// vs
Path.Combine("/", "")

results in an empty path but it should just be root.

Attempting to gather dependency information for package 'Zio.0.1.0' with respect to project 'Xamarin.HealthVault.UX.Prototoyping', targeting '.NETPortable,Version=v4.5,Profile=Profile111'
GET https://api.nuget.org/v3/registration1-gz/zio/index.json
OK https://api.nuget.org/v3/registration1-gz/zio/index.json 438ms
Total number of results gathered : 1
Gathering dependency information took 518.52 ms
Summary of time taken to gather dependencies per source :
https://api.nuget.org/v3/index.json - 475.58 ms
Attempting to resolve dependencies for package 'Zio.0.1.0' with DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'Zio.0.1.0'
Resolved actions to install package 'Zio.0.1.0'
Retrieving package 'Zio 0.1.0' from 'nuget.org'.
For adding package 'Zio.0.1.0' to project 'Xamarin.HealthVault.UX.Prototoyping' that targets 'portable45-net45+win8+wpa81'.
Install failed. Rolling back...
Package 'Zio.0.1.0' does not exist in project 'Xamarin.HealthVault.UX.Prototoyping'
Package 'Zio.0.1.0' does not exist in folder '/Users/wilvoss/Projects/DefaultCollection/Health/_git/health-design/Prototypes/UX/SleepyTime/SleepyTime/packages'
Executing nuget actions took 200.59 ms
Could not install package 'Zio 0.1.0'. You are trying to install this package into a project that targets '.NETPortable,Version=v4.5,Profile=Profile111', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.

Currently it only allows mounts under root (no subfolders). This requires a bit more (careful) work to the way paths are mapped to filesystems.

Problem

EnumeratePaths via AggregateFileSystem is searching folder by folder when using SearchOption.AllDirectories

Description

I've been hitting a problem in my codebase for a while where doing a simple EnumeratePaths on a AggregateFileSystem with a couple of Physical subsystems was very slow, taking just over 10 seconds to find 8000 files on my Samsung 980 NVME SSD.

This is one of the folders I'm enumerating over.

So obviously I thought this is probably pretty expected and was about to write a bunch of caching stuff, but when looking at the code I feel like way AggregateFileSystem is doing stuff here is bad for everyone.

Expected

I expected it to call EnumeratePaths on all of the child filesystems and return a list of distinct results. There's probably a good reason it works this way instead, but I can't think of it.

I tried BinaryFormatter, but failed. It said Type 'Zio.FileSystems.MemoryFileSystem' in Assembly 'Zio, Version=0.7.4.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.

While it might not be a bug these filesystems use a catch all search pattern and handle recursion themselves. This removes opportunities for optimizations based on the query, such as having a special case when searching for files with a specific extension, which can be a common operation.

In the MemoryFileSystem, when enumerating paths, the contents of the SortedSet<UPath> are returned using a foreach with a yield instead of simply returning the SortedSet<UPath> itself. Any reason for this?

Is there any plans to allow the implementation of Async File Systems, EG: Azure Blob?

I'm using a physical file system and a subsystem and if I'm using OSX it works perfectly fine, but on Windows it fails with 'Path 'c:/path/here' must be absolute'

var path = "c:/path/here/"
fs = new PhysicalFileSystem();
subfs = new SubFileSystem(fs, path);

This also fails on slower 'C:\path\here\'

I thought it was the trailing slash removal until I followed and noted that AssertAbsolute throws even with the trailing slash since the path does not start with / but c:/

As the search under windows, I expect the * to match everything in the given path. So if I create a searchPattern like this:

var path = new UPath("/folder1");
var search = "*";
var pattern = SearchPattern.Parse(ref path, ref search);

I wouldn't expect it to match a path not beginning with folder1, like /folder2/test.

Is that intended behaviour or a bug? Because the code states the scenario of having the * as the search pattern as an optimized and most common case. I agree that it is the most common case, but only in combination with the pre-defined path not being ignored.

It just prepends root path to the relative path without considering current working directory.

Can you implement this two features?

1. Add member for current working directory to SubFileSystem
2. Add method for changing the member mentioned in 1.

Thank you.