bechtleav360 / tusdotnet.storage.s3

Extension to support S3 as a storage backend for the tusdotnet TUS implemntation

tusdotnet.Stores.S3

The Package tusdotnet.Stores.S3 implements all necessary interfaces to use S3 as a file storage backend. The Implementation is put into the TusS3Store class

What is tus?

Tus is a web based protocol for resumable uploads. Implementations and client libraries exist for many platforms.

What is tusdotnet?

tusdotnet is a popular implementation of the tus protocol for .net.

Why do I need tusdotnet.Stores.S3?

Tusdotnet only comes with a disk storage implementation. This extension allows you to use s3 blobstorage instead of local (or network attached) disk.

Implemented Extensions

The tus protocol offers a few extensions. The following extensions are implemented:

• Termination - Allows for deletion of completed and incomplete uploads.
• Expiration - Server will consider uploads past a certain time expired and ready for deletion.
• Pipelines - more efficient than handling plain streams

Configuration

In order to allow this backend to function properly, the user accessing the bucket must have at least following AWS IAM policy permissions for the bucket and all of its subresources:

s3:AbortMultipartUpload
s3:DeleteObject
s3:GetObject
s3:ListMultipartUploadParts
s3:PutObject

While this package uses the official AWS SDK for Go, S3Store is able to work with any S3-compatible service such as MinIO. In order to change the HTTP endpoint used for sending requests to, adjust the BaseEndpoint option in the AWSSDK.S3 nuget package (https://aws.amazon.com/sdk-for-net/).

Implementation

Once a new tus upload is initiated, multiple objects in S3 are created:

First of all, a new info object is stored which contains a JSON-encoded blob of general information about the upload including its size and meta data. This kind of objects have the suffix ".info" in their key.

In addition a new multipart upload (http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html) is created. Whenever a new chunk is uploaded to tus using a PATCH request, a new part is pushed to the multipart upload on S3.

If meta data is associated with the upload during creation, it will be added to the multipart upload and after finishing it, the meta data will be passed to the final object.

Once the upload is finished, the multipart upload is completed, resulting in the entire file being stored in the bucket. The info object, containing meta data is not deleted. It is recommended to copy the finished upload to another bucket to avoid it being deleted by the Termination extension.

If an upload is about to being terminated, the multipart upload is aborted which removes all of the uploaded parts from the bucket. In addition, the info object is also deleted. If the upload has been finished already, the finished object containing the entire upload is also removed.

Considerations

In order to support tus' principle of resumable upload, S3's Multipart-Uploads are internally used.

In addition, it must be mentioned that AWS S3 only offers eventual consistency (https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel). Therefore, it is required to build additional measurements in order to prevent concurrent access to the same upload resources which may result in data corruption.

Usage

ILogger<TusS3Store> logger;

var tusS3StoreConfig = new TusS3StoreConfiguration()
{
    BucketName = options.Value.BucketName
};

var awsCredentials = new BasicAWSCredentials("myaccessKey", "mysecretkey");

AmazonS3Config s3Config = new AmazonS3Config
{
    // MUST set this before setting ServiceURL and it should match the `MINIO_REGION` environment variable
    AuthenticationRegion = "us-east-1",
    // MUST be true to work correctly with MinIO server
    ServiceURL = "https://mys3endpoint.com",
    ForcePathStyle = true,
};

var tusStore = new TusS3Store(logger, tusS3StoreConfig, awsCredentials, s3Config);

var tusConfig = new DefaultTusConfiguration
{
    Store = CreateTusS3Store(services),
    MetadataParsingStrategy = MetadataParsingStrategy.AllowEmptyValues,
    UsePipelinesIfAvailable = true,
    // Set an expiration time, where incomplete files can no longer be updated.
    // This value can either be absolute or sliding.
    // Absolute expiration will be saved per file on create
    // Sliding expiration will be saved per file on create and updated on each patch/update.
    Expiration = new SlidingExpiration(TimeSpan.FromMinutes(1))
};

Special Thanks

• Bechtle A/V Software Solutions 360° for sponsoring the project
• Xtensible.TusDotNet.Azure for a boilerplate to create this extension

tusdotnet.storage.s3's Issues

Injecting IAmazonS3 client in TusS3Store constructor

Hi, I would like to know if there's any method to inject IAmazonS3 client on TusS3Store instead of specifying the credentials (AWS Api Key & Secret).

S3 GetMetadataAsync get null metadata although the metadata is written to s3 object.

Hi, I observed that is an issue when trying to GetMetadataAsync from the file, it returns nothing, the root cause is this function when joining the string of metadata collection returns from s3.

There's a spacing after the comma which caused the MetadataParser unable to parse the metadata.
This is the following constructed metadata string:
x-amz-meta-filename a2V5Y2xvYWstMjMuMC42LnppcA==, x-amz-meta-filetype YXBwbGljYXRpb24veC16aXAtY29tcHJlc3NlZA==, x-amz-meta-relativepath bnVsbA==

Error message that thrown by MetadataParser.
Header Upload-Metadata: The Upload-Metadata request and response header MUST consist of one or more comma - separated key - value pairs. The key and value MUST be separated by a space.The key MUST NOT contain spaces and commas and MUST NOT be empty. The key SHOULD be ASCII encoded and the value MUST be Base64 encoded. All keys MUST be unique. The value MAY be empty. In these cases, the space, which would normally separate the key and the value, MAY be left out.

While I checked and compared with the original local disk metadata string, it looks like this and no problem of parsing it.
relativePath bnVsbA==,name a2V5Y2xvYWstMjMuMC42LnppcA==,type YXBwbGljYXRpb24veC16aXAtY29tcHJlc3NlZA==,filetype YXBwbGljYXRpb24veC16aXAtY29tcHJlc3NlZA==,filename a2V5Y2xvYWstMjMuMC42LnppcA==

the S3 storage key prefix should be configurable

Hi, first of all, thanks a lot for taking the time to implement this library!

The TusS3Api class currently implements the calculation of the S3 key as follows:

    private string GetFileKey(string key)
    {
      string str = "files/";
      if (!string.IsNullOrWhiteSpace(str) && !str.EndsWith("/"))
        str += "/";
      return str + key;
    }

would it be possible to make the hardcoded "files/"prefix configurable? We have an already existing directory structure in place that I would like to connect this store to, the logic of that structure can nicely be encoded in an implementation of ITusFileIdProvider, except this prefix... Happy to submit a PR if you agree...