A faster collectstatic command.
Features
- Efficiently decide what files to upload using cached checksums
- Parallel file uploads
Supported Storage Backends
storages.backends.s3boto3.S3Boto3Storage
storages.backends.gcloud.GoogleCloudStorage
django.core.files.storage.FileSystemStorage
Running Django's collectstatic
command can become painfully slow as more and
more files are added to a project, especially when heavy libraries such as
jQuery UI are included in source code. Collectfast customizes the builtin
collectstatic
command, adding different optimizations to make uploading large
amounts of files much faster.
Install the app using pip:
$ python3 -m pip install Collectfast
Make sure you have this in your settings file and add 'collectfast'
to your
INSTALLED_APPS
, before 'django.contrib.staticfiles'
:
STATICFILES_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
COLLECTFAST_STRATEGY = "collectfast.strategies.boto3.Boto3Strategy"
INSTALLED_APPS = (
# ...
"collectfast",
)
Note: 'collectfast'
must come before 'django.contrib.staticfiles'
in
INSTALLED_APPS
.
Note: The boto strategy will set preload_metadata
on the remote storage
to True
, see #30.
Collectfast Strategy | Storage Backend |
---|---|
collectfast.strategies.boto3.Boto3Strategy | storages.backends.s3boto3.S3Boto3Storage |
collectfast.strategies.gcloud.GoogleCloudStrategy | storages.backends.gcloud.GoogleCloudStorage |
collectfast.strategies.filesystem.FileSystemStrategy | django.core.files.storage.FileSystemStorage |
Custom strategies can also be made for backends not listed above by
implementing the collectfast.strategies.Strategy
ABC.
Collectfast overrides Django's builtin collectstatic
command so just run
python manage.py collectstatic
as normal.
You can disable Collectfast by using the --disable-collectfast
option or by
setting COLLECTFAST_ENABLED = False
in your settings file.
It's recommended to setup a dedicated cache backend for Collectfast. Every time
Collectfast does not find a lookup for a file in the cache it will trigger a
lookup to the storage backend, so it's recommended to have a fairly high
TIMEOUT
setting.
Configure your dedicated cache with the COLLECTFAST_CACHE
setting:
CACHES = {
"default": {
# Your default cache
},
"collectfast": {
# Your dedicated Collectfast cache
},
}
COLLECTFAST_CACHE = "collectfast"
If COLLECTFAST_CACHE
isn't set, the default
cache will be used.
Note: Collectfast will never clean the cache of obsolete files. To clean
out the entire cache, use cache.clear()
. See docs for Django's cache
framework.
Note: We recommend you to set the MAX_ENTRIES
setting if you have more
than 300 static files, see #47.
The parallelization feature enables parallel file uploads using Python's
concurrent.futures
module. Enable it by setting the COLLECTFAST_THREADS
setting.
To enable parallel uploads, a dedicated cache backend must be setup and it must use a backend that is thread-safe, i.e. something other than Django's default LocMemCache.
COLLECTFAST_THREADS = 20
By default, Collectfast will suppress any exceptions that happens when copying
and let Django's collectstatic
handle it. To debug those suppressed errors
you can set COLLECTFAST_DEBUG = True
in your Django settings file.
Please feel free to contribute by using issues and pull requests. Discussion is open and welcome.
The test suite is built to run against live S3 and GCloud buckets. You can
disable live tests by setting SKIP_LIVE_TESTS=true
or running make test-skip-live
. To run live tests locally you need to provide API credentials
to test against. Add the credentials to a file named storage-credentials
in
the root of the project directory:
export AWS_ACCESS_KEY_ID='...'
export AWS_SECRET_ACCESS_KEY='...'
export GCLOUD_CREDENTIALS='{...}' # Google Cloud credentials as JSON
Install test dependencies and target Django version:
python3 -m pip install -r test-requirements.txt
python3 -m pip install django==2.2
Run test suite:
make test
Code quality tools are broken out from test requirements because some of them only install on Python >= 3.7.
python3 -m pip install -r lint-requirements.txt
Run linters and static type check:
make lint
Collectfast is licensed under the MIT License, see LICENSE file for more information.