xi-project / xi-filelib

File library for PHP 5.5

Xi Filelib

Filelib is a file library component for PHP, providing a virtual filesystem for your web application's files. It can be used to manage both your application's internal and user-uploaded files in many ways. What it (at least for now) is NOT is a place to store your assets like css, js and similar.

Let's face it: practically all web apps have to store documents, media and such and the needs are same. Filelib takes care of all the hard and/or repetetive tasks and abstracts away and reveals all the related changeable components as loosely coupled subsystems:

• Storing of metadata (virtual folders / files) in a data storage (database, usually).
• Safe physical storage of files in a filesystem (local, S3, Gridfs, etc).
• Authorization (who can do what in the virtual filesystem) and publication (to make them fast-readable and with pretty urls) of world readable files.
• Rendering the files to HTTP response (sometimes you can't world-publish everything but it's still gotta be decent).
• Mime types and extensions and all this horrible stuff
• Versioning (meaning creation of different representations of a master file, thumbnails, html5 videos etc)
• Asynchronous processing (you can't keep the end user waiting for that video to be encoded, you know)
• And there's probably more!

Filelib is fully extensible via plugins and hooks. In fact, many of the "core" functionality is provided via plugins (authorization, automatic publishing, file versions) so the core is kept elegant and maintainable.

Filelib is based on my own observations, opinions and experience formed while developing many file- and mediabanks for the last 10 years. It has evolved and keeps evolving with real projects and use cases, so thanks for all past and present early adopters!

Project Status

(2018-08) I've kinda moved on years ago. Not coding much with PHP anymore. Doing React and stuff with JS nowadays. It's a shame, because Filebanksta was soooo close to be "finished". It remains useable, though. I just upgraded all packages to modern versions. I'll try to do it again a couple times a year, at least. Cheers!

Hard requirements

• A client software that needs file management
• PHP 7.1

Soft requirements (for harder use)

• A serious data storage (MySQL/MariaDB, PostgreSQL, MongoDB supported out of the box)
• Imagemagick for image processing
• Zencoder for all your video needs
• Intl for transliterating / slugifying
• A queue (RabbitMQ, IronMQ, SQS) for asynchronous operations

Quickstart

Using JSON storage (for simple testing only)

<?php

use Xi\Filelib\FileLibrary;
use Xi\Filelib\Backend\Adapter\JsonBackendAdapter;
use Xi\Filelib\Storage\Adapter\FilesystemStorageAdapter;
use Pekkis\DirectoryCalculator\DirectoryCalculator;
use Pekkis\DirectoryCalculator\Strategy\UniversalLeveledStrategy;

$filelib = new FileLibrary(
    new FilesystemStorageAdapter(
        __DIR__ . '/files',
        new DirectoryCalculator(new UniversalLeveledStrategy()
    ),
    new JsonBackendAdapter(__DIR__ . '/filelib-example.json')
);

$file = $filelib->uploadFile('/path/to/some/file.jpg');

// Display the image
header("Content-Type: " . $file->getMimetype());
echo file_get_contents($filelib->getStorage()->retrieve($file->getResource()));

Documentation

TODO!

Examples and use cases

https://github.com/xi-project/xi-filelib-examples contains a lot of examples from very simple to very complex use cases. Clone the repo, configure a web server and dive straight into the code!

About integrating to your own software

Experience has time and again proved that integration should be light. If one is using Doctrine ORM, it could seem appropriate to integrate via Filelib's entities. DON'T DO IT. It will bite you back.

All the stuff inside the backend platforms (and other deeper abstractions provided by Filelib) are the library's private parts and subject to change any time. So if you utilize these internals, be prepared to enter upgrade hell at some point. For example the entities in the ORM backend platform; they may go away for good some day.

Just utilize Filelib's identifiables' (folders / files) ids / UUIDs within your own data and domain and use functionality provided by Filelib for everything else.

FolderRepository and FileRepository should usually be as deep as you have to go but it may not yet be the case. For some "tougher" operations I've personally had to use Backend and ProfileManager, at least. These are things and use cases that are yet to be considered before 1.0 is reached.

Framework integration

For framework integration, see:

• https://github.com/xi-project/xi-bundle-filelib (Symfony bundle, needs a maintainer, I'm a very bad one!)
• https://github.com/xi-project/xi-filelib-symfony-sandbox (Symfony sandbox, needs a maintainer too!)

Actual applications using Filelib

For actual applications I know are using Filelib, see:

• http://autot.oikotie.fi: Classified ads for cars. Ancient version if not upgraded without my knowledge / help.
• https://oy.eautokoulu.fi: Learning environment. Advanced large-scale usage of both images and videos. Current version.
• http://sykettatyohon.fi: Document / image managent provided by probably a somewhat older version.

Know more? Please tell!

xi-filelib's Issues

Sanitize event names

Follow the conventions (lowercase, underline). Namespace with "xi_filelib".

The UUID problem

Files and folders should have an id independent of the backends because of the asynchronous operations must return something that can be used to find objects later on. It was decided to implement UUID.

• Interface for UUID creation (there are many implementations such as pure PHP, PECL package).
• Implementations (pure PHP, PECL, what else)
• Integration of UUID to the commands and methods of Filebanksta
• Implementation of UUID to the objects (file, folder, resource???)

Uploading the same file twice should not take twice the space

Introduce some kind of hashing for the files to know which files are the same. There's no need to store the same file many times.

Missing file suffix

When uploading for example pdf files and later displaying the url for it, the file suffix is missing.

Put event names into constants

Event names are strings. Make them into consts in FileLibrary or perhaps FileLibrary\Events.

Storage's retrieve works wrong

Works with holy spirit only.

Remove dependency on Doctrine ORM. Support PDO and (probably) DBAL.

They will probably share code or even be the same class.

Missing Documentation and/or Use Cases

Hi,
This project looks great, I've scanned through and can see how it could be very useful.
However, it is lacking in documentation or use cases.
Plus a summary of how it can help.

Is this something in the pipeline?

Rename DB columns

This is an unfortunate BC break, but I think it must be done before 1.0.

I think columns should be renamed in the following way:

xi_filelib_folder:
    foldername -> name
    folderurl -> url

xi_filelib_file:
    fileprofile -> profile
    filesize -> size
    filename -> name
    date_uploaded -> created

Same thing in MongoDB backend. Migrations should of course be provided.

Make a generic DirectoryIdCalculator the default

Phase 1

Make TimeDirectoryIdCalculator the default
It works with both MongoDB and SQL DBs. The directory structure it generates is more intuitive to the user.

Phase 2

Make a generic DirectoryIdCalculator that works with any DB and creates smooth distributions. (unlike TimeDirectoryIdCalculator)

---

Consider naming the files as <id>.<extension> where <extension> is inferred from the mimetype. Store the extension in the DB in case the mimetype-extension map changes later. Now the default directory structure is easy to understand.

Logging should be introduced

Filebanksta needs logging.

Move tests under Xi\Filelib\Tests

The namespace for the tests is currently Xi\Tests\Filelib but it should be Xi\Filelib\Tests.

Exceptions should be refactored

At the moment filebanksta throws random exceptions, mostly FilelibExceptions. Everything should throw good exceptions, falling back on a marker interface (FilelibException).

Use Xi Collections

Methinks Filelib could use Xi collections. To support that project and for great success in general!

S3 publisher

We have a S3 storage which is kind of dum. What we should have is S3 / CloudFront publisher.

Rename Doctrine2Backend to DoctrineORMBackend

Doctrine2Backend should be renamed to DoctrineORMBackend to allow other Doctrine backends too. For example DoctrineODMMongoDBBackend.

Immutable version support

ATM, original files are not versions. Consider adding support for immutable versions and making the original one.

Prune issues

pekkis and hype- should go through the issues and

• prune deprecated, duplicate and silly issues
• clarify issues so that they provide sufficient information
• add known issues and assignations

Commands v2

Commands and tests must be refactored. I now see errors in my ways.

Fix inceptions

Stuff work the wrong way. Write use case instructions and integration tests. Add documentation. Fix.

AbstractVersionProvider has methods which are not on VersionProvider interface

Xi\Filelib\Plugin\VersionProvider\AbstractVersionProvider has a lot of methods which are not on Xi\Filelib\Plugin\VersionProvider\VersionProvider interface.

EventDispatcher for FileLibrary as on _optional_ argument

FileLibrary constructor parameter EventDispatcherInterface $eventDispatcher as nullable / optional.

Bug: filesize is not working

Uploading any file will result in filesize 0 in database. Tested with Doctrine2Backend and MySQL.

New backend / storages don't create root folder

It seems no one creates the root folder any more when it's missing. All kind of errors.

Add PostgreSQL to travis CI

PostgreSQL should be added to Travis CI to make sure it works.

Introduce caching for backends

The backend operations are the slowest ones, so they could use some caching. The software should also be profiled and see where the other bottlenecks are.

Is there any reason to reinvent the wheel or should we use for example the Doctrine\Common\Cache?

SymlinkFilesystemPublisher : setRelativePathToRoot has wrong return value

 * @return SymlinkPublisher  

=> SymlinkFilesystemPublisher

also function should return $this;

Associate mime types to über type

ATM it's a kludge. Mimetype is split and first part is returned...

Rethink base FileLibrary object

• Common use cases (findFile, findFolder, etc)
• Configuration object of some kind / constructor params.
• What else?

Interface check

Interfaces have too many setters in them. Check 'em and remove.
Interfaces (like FileOperator) may have fallen behind. Check.

Backend (at least) needs fancier architecture

Backend needs at least these:

• Some kind of an abstract finder (no own query language)
• Identity map
• Unit of work, maybe (clean / dirty states)

Consider replacing FileProfile with FileMatcher

Consider removing FileProfile and introducing the following interface:

public function addPlugin(Plugin $plugin, callable $matcher = null);

$filelib->addPlugin($myPlugin1, Matchers::getDirectoryMatcher('catpics'));
$filelib->addPlugin($myPlugin2, function (File $file) { return $file->getDirectory() == '/catpics'; });
$filelib->addPlugin($myPlugin3); // Plugin applied to all files

JSON platform

Will make a simple JSON platform for test / dev use. No need for Doctrine in the Hello World.

Refactor Fileupload

Fileupload extends fileobject which extends SPLFileObject.

Fileupload should contain fileobject because SPLFileobject can't be serialized and it makes stuff hard.

Backends should throw InvalidArgumentException

SQL backends should throw InvalidArgumentException when given a non-numeric value to a findXxx method.

Rename VersionPlugin -> ImageMagick[Version]Plugin

Make unsettable values unsettable

For example identifiables' setId() must go. We will use reflection, I already have investigated Doctrine ORM for guidance and tested stuff.

Make operations atomic

Operations should be atomic. For example a file deletion should be rolled back if either metadata or physical file deletion fails.

Filelib needs copy functionality

as a user I need the possibility to copy files (another instance) because I can't use the same instance if the base file changes

Simplify file and folder stuff

One implementation should be enought. The need to change implementation class is ancient and the new hooks and stuff make it completely obsolete.

Xi\Tests\Filelib\Publisher\Filesystem\TestCase uses obsolete namespace

use Xi\Filelib\Publisher\Filesystem\SymlinkPublisher;
should be:

use Xi\Filelib\Publisher\Filesystem\SymlinkFilesystemPublisher;

Make a pure PHP renderer

Want a default implementation that does not depend on Symfony.

Stuff in wrong namespaces

Don't like where stuff are. Move and groove. File, Folder, etc may as well be in the base namespace. Grr.

Separate ZF1 stuff to their own package

End of life for ZF1.

Setter abuse must go

For historical and stupidness reasons, constructors have too few parameters and classes too many setters. Refactor.

OBS! This affects both the S2 bundle and ZF integration. Which, by the way, has already fallen behind.

Get rid of the Configurator

While removing the inceptions and writing use cases, will get rid of the Constructor brainfart pattern from the Zend days.

public function __construct(array $options)
{
Configurator::setOptions($this, $options)
// Delegates to setters.
}

Refactor stuff in wrong namespaces / places

There are some semantic errors in namespacing / naming / placing of classes. Refactor.

OBS! Keep stuff inside namespace.

NOT LIKE THIS: Xi\Filelib\File

LIKE THIS: Xi\Filelib\File\File

Yes i know it's a little retarded but hey, it was chosen to keep namespaced stuff in their OWN namespace, so that's how it's gonna go down.

Queue 2.0

Version 2 of queueing: topicked queueing and topicked processors for more fine-grained async ops.

Advanced interface (TopicQueue, f.ex).

Decide the fate of integration packages

Symfony bundle and ZF integration stuff needs tests and maintenance.

Move stuff from Core to Plugins

Identify and extract core components that could simply be plugins.

Found so far:

• Publisher and Acl. Provide (a base class for) a plugin that can do publishing (when it decides to).
• Limiter