consilience / flysystem-azure-file-storage

Flysystem adapter for Windows Azure File Storage

Azure File Storage adapter for Flysystem

This repo is fork of League\Flysystem\Azure with the underlying Azure API library changed from microsoft/azure-storage to microsoft/azure-storage-file. The original driver supports Azure blob storage, with a flat binary object structure. This driver supports Azure file storage, which includes directory capabilities.

A separate service provider package for Laravel 5.5+ is available here: https://github.com/academe/laravel-azure-file-storage-driver The service provider allows Azure File Storage shares tbe be used as a native filesystem within Laravel.

Installation

Install package

composer require consilience/flysystem-azure-file-storage

How to use this driver

Note: if you are using Laravel then the filesystem driver will wrap and abstract all of this for you.

use League\Flysystem\Filesystem;
use Consilience\Flysystem\Azure\AzureFileAdapter;
use MicrosoftAzure\Storage\File\FileRestProxy;
use Illuminate\Support\ServiceProvider;

// A helper method for constructing the connectionString may be usedful,
// if there is a demand.

$connectionString = sprintf(
    'DefaultEndpointsProtocol=https;AccountName=%s;AccountKey=%s',
    '{storage account name}',
    '{file storage key}'
);

$config = [
    'endpoint' => $connectionString,
    'container' => '{file share name}',
    // Optional to prevent directory deletion recursively deleting
    // all descendant files and direcories.
    //'disableRecursiveDelete' => true,
    // Optional driver options can also be added here. e.g. CacheControl, Metadata.
];

$fileService = FileRestProxy::createFileService(
    $connectionString,
    [] // $optionsWithMiddlewares
);

$filesystem = new Filesystem(new AzureFileAdapter(
    $fileService,
    $config,
    'optional-directory-prefix'
));

// Now the $filesystem object can be used as a standard
// Flysystem file system.
// See https://flysystem.thephpleague.com/api/

// A few examples:

$content    = $filesystem->read('path/to/my/file.txt');
$resource   = $filesystem->readResource('path/to/my/file.txt');
$success    = $filesystem->createDir('new/directory/here');
$success    = $filesystem->rename('path/to/my/file.txt', 'some/other/folder/another.txt');

// The URL of a file can be found like this:

$url = $filesystem->getAdapter()->getUrl('path/to/my/foo.bar');

Testing

Set up .env and run live tests:

composer install
vendor/bin/phpunit --testsuite flysystem-azure-live-tests

These will create/delete a few test files and directories in the root of the Azure file share.

flysystem-azure-file-storage's Issues

Upgrade for Laravel 9.x (PHP 8.1, Flysystem 3.0)

Laravel 9.x uses PHP 8 and Flysystem 3. A two-stage upgrade should take us to PHP 8.1 first, then Flysystem 3.

• New Flysystem API.
• PHP 8.1 (dependencies and a few tweeks).
• Support new prefix method (League\Flysystem\PathPrefixer object).
• New Flysystem test suite (needs mocking of Azure classes, pretty involved; though maybe not if it actually runs against a live file share).
• Consistently apply all exceptions.
• Handle visibility.
• Update the docs, especially example results.

Streams not quite working correctly

Was trying to copy a file between two separate disks on Laravel:

        $destinationFileSystem->writeStream(
            $filename,
            $sourceFileSystem->readStream($filename)
        );

Error from Azure is:

Code: 400
Value: An HTTP header that's mandatory for this request is not specified.
details (if any): <?xml version="1.0" encoding="utf-8"?><Error><Code>MissingRequiredHeader</Code><Message>An HTTP header that's mandatory for this request is not specified.

Workaround in my application for now:

        $destinationFileSystem->put(
            $filename,
            stream_get_contents($sourceFileSystem->readStream($filename))
        );

My files are limited in size, so this is not such a problem. It would still be nice to get to the bottom of this issue though, albeit not with much urgency.

Given that workaround, the stream reading is working, but the stream writing is not.

This would be a simpler line than the stream reading, but I've left it in as a reminder:

$sourceFileSystem->read($filename)

Move to consilience organisation

The vendor is consilience so it makes sense to move the repo. There were technical reasons why this could not be done initially.

Moving the vendor on packagist is going to be an important part of this move.

Not listing directories

getContents() is returning files, but no directories.

[1.x]: Excessive client calls

Problem
In version 1.x of this package does a separate metadata call for each file and directory found in a directory content listing.

Additional information
Since the update to version 1.x I started to receive rate limit warnings from Azure. This turned out to be from the listContents method call used on Flysystem using the Azure Adapter. This methods uses the getContents call https://github.com/consilience/flysystem-azure-file-storage/blob/1.0.0/src/AzureFileAdapter.php#L407 in the AzureFileAdapter. In the getContents a request is done to retrieve the listDirectoriesAndFiles method from the client to retrieve a list from the Azure File Storage. This returns a list of all the directories and files and their sizes. On version 0.1.6 and before of the package this was the only information used. https://github.com/consilience/flysystem-azure-file-storage/blob/0.1.6/src/AzureFileAdapter.php#L456

In 1.0 this information is expanded with additional meta information retrieved by calling getFileProperties and getDirectoryProperties for each file or directory found https://github.com/consilience/flysystem-azure-file-storage/blob/1.0.0/src/AzureFileAdapter.php#L483. This generates a lot of calls quickly. Their is the first call (listDirectoriesAndFiles). If their are for example 3 directories and 60 files, then the total would be 1 + 3 + 60 = 64 calls just to retrieve the list of contents of a directory.

Possible solution
Not retrieving the additional information for files (and directories). This forces the user to retrieve the additional information if necessary, but I would estimate that the additional information besides the filename is not always used and therefore removing the need to have the information available by default.

In a project it was for now mitigated by patching the file with the following code:

diff --git a/vendor/consilience/flysystem-azure-file-storage/src/AzureFileAdapter.php b/vendor/consilience/flysystem-azure-file-storage/src/AzureFileAdapter.php
--- a/vendor/consilience/flysystem-azure-file-storage/src/AzureFileAdapter.php
+++ b/vendor/consilience/flysystem-azure-file-storage/src/AzureFileAdapter.php
@@ -477,14 +481,11 @@

         foreach ($listResults->getFiles() as $fileObject) {
             $pathName = trim(sprintf('%s/%s', $path, $fileObject->getName()), '/');
-
-            $contents[] = $this->normalizeFileProperties(
-                $pathName,
-                $this->azureClient->getFileProperties(
-                    $this->container,
-                    $this->prefixer->prefixPath($pathName),
-                ),
-            );
+            $contents[] = FileAttributes::fromArray([
+                'type' => 'file',
+                'path' => $pathName,
+                'dirname' => dirname($pathName),
+            ]);
         }

         return $contents;

Exception when using directory prefix

When creating a file using a prefix, the file uploads, but then the following exception is thrown:

Code: 404
Value: The specified parent path does not exist.
details (if any): <?xml version="1.0" encoding="utf-8"?><Error><Code>ParentNotFound</Code><Message>The specified parent path does not exist.

Unit tests and integration tests

Moving files does not work for non-URL safe paths or filenames

Issue from PR #1 (@maetthu sorry, I had issues disabled for some reason)

Copying/Renaming files containing spaces, unicode characters or other non-url-safe characters results in following error:

Fail: Code: 403 Value: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. details (if any): AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. [...] The MAC signature found in the HTTP request 'XXX' is not the same as any computed signature.

Uploading files to root folder w/o path prefix results in a similar error.

Problem with uploading files

I'm able to list files and read file contents from Azure File Storage, so I'm pretty sure my connection string is fine.
When I try to write/update a file, I get the following error:

403: Fail: Code: 403 Value: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. details (if any): <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:33698f74-a01a-005c-7ab3-6f6a14000000 Time:2020-08-11T07:48:02.1835762Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'l+Lz9lBDaKq9VZ3k9zoBozQ5j9prREu0VkO9i/LxzxU=' is not the same as any computed signature. Server used following string to sign: 'GET application/x-www-form-urlencoded Tue, 11 Aug 2020 07:48:02 GMT x-ms-request-id:5f324d322c9eb x-ms-version:2016-05-31 /sjlmsib/s3ib/smarthub_sync/new/ restype:directory'.</AuthenticationErrorDetail></Error>.

I'm able to write to the file using the same connection string, but Microsoft API's createFileFromContent function, so I'm sure that my access rights allow me to create and write the destination file.

For reference, this works (it creates the file in the Azure File Share)
$microsoftFileClient->createFileFromContent('share', 'test.csv', $testdata);

And this doesn't work (it produces the error above)
$azureFileAdapterClient->write('test.csv', $testdata);

Any way to fix?

Make URL for a file public

Issue raised here:

academe/laravel-azure-file-storage-driver#1

Cannot stream from one share into another

I'm sure this used to work, but now throws "missing header" exceptions at the HTTP level communicating with Azure. That threw me quite a lot.

It appears that the read stream the API provides no longer can be stated, i.e. details about its length is not available. When writing a stream to the API, it expects to know the content length from the start. The MS library was assuming the length was available just because a resource was provided, when in fact the length was unknown - null. This caused the request to the stream writing to the API to be corrupted - it had a null length. And that was throwing up all sorts of obscure errors.