antonagestam / collectfast Goto Github PK
View Code? Open in Web Editor NEWA faster collectstatic command.
Home Page: https://pypi.org/project/Collectfast/
License: MIT License
A faster collectstatic command.
Home Page: https://pypi.org/project/Collectfast/
License: MIT License
Is there a simple way to purge cached values? I by mistake added a bunch with no expiration to the wrong cache and want to clear them.
Django is 1.10.7,
Traceback (most recent call last):
File "manage.py", line 29, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 208, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 41, in load_command_class
return module.Command()
File "/usr/local/lib/python3.5/site-packages/collectfast/management/commands/collectstatic.py", line 35, in __init__
if self.storage.preload_metadata is not True:
File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 235, in inner
return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute 'preload_metadata'
Traceback (most recent call last):
File "", line 17, in
File "/home/abal/isocutter/build/collectfast/setup.py", line 10, in
long_description=open('README.rst').read(),
File "/home/abal/isocutter/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 12: ordinal not in range(128)
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 17, in
File "/home/abal/isocutter/build/collectfast/setup.py", line 10, in
long_description=open('README.rst').read(),
File "/home/abal/isocutter/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 12: ordinal not in range(128)
trying out collectfast with django-minio-storage, I stumbled over collectfast.etag.get_remote_etag() causing an
AttributeError: 'MinioStaticStorage' object has no attribute '_normalize_name'
which can easily be replaced by the public storages.utils.safe_join() which is called by storages.backends.s3boto3.S3Boto3Storage._normalize_name()
Running pip install collectfast fails because the long description is set to
long_description=open('README.md').read()
but README.md isn't included in the package.
I realize its not a collectfast problem, but I spent a long time trying to figure out why caching was only working for a (seemingly random) portion of my static files. Eventually I found the MAX_ENTRIES default setting for Django caches, which is set to 300 by default (I think this only applies to DB or File Based caching, which is what i'm using for my CollectFast cache).
It may be useful for some users to call out the bad default on the "MAX_ENTRIES" option when you call out "TIMEOUT" in the README, since they probably won't be familiar with it and it will almost always reduce the effectiveness of CollectFast if used with affected cache types, since only 300 file hashes will be cached
There should be an option to cache the ETag of the remote files to further decrease deploy time.
There's an informational logging message in the etag implementation:
collectfast/collectfast/etag.py
Line 120 in 8bcdd77
This is shown if collectstatic is run with verbosity 0 and should not be. This can be fixed in the management command by using logging.disable() if verbosity is zero.
I use S3 for production, but on my dev machine I use STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
.
If I have Collectfast
installed on my dev machine I get this traceback:
./manage.py collectstatic --noinput
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
return self.handle_noargs(**options)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 147, in handle_noargs
collected = self.collect()
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 39, in collect
ret = super(Command, self).collect(*args, **kwargs)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 111, in collect
handler(path, prefixed_path, storage)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 80, in copy_file
normalized_path = self.storage._normalize_name(path)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/utils/functional.py", line 214, in inner
return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute '_normalize_name'
The readme.rst contains non-ascii chars which fails the setup on python 3 if i install it via pip.
It would work if I do open with encoding='utf-8'
, but that would fail python 2 i assume (I haven't used python 2 since years)?
Hello!
When using collectfast with threads, there is a problem with call_command:
from django.core.management import call_command
def update():
call_command('collectstatic', interactive=False, verbosity=3)
return 0
This immediately return 0 and running threads are killed.
We need to make sure that collectstatic is a blocking call until all work is done.
Greets
Manuel
Collectfast only ever makes a single cache entry on Windows, meaning I can't collectstatic
from my development machine.
I believe I have traced this to the use of storage._normalize_name(prefixed_path)
, which returns a Windows style directory structure with double backslashes instead of forward slashes, i.e. it doesn't normalize paths.
I'll see if there's anything I can do to patch it and I'll create a pull request, unless someone come up with a better place to fix it.
I apologize in advance if this isn't an issue directly related to this library, but I can't figure out what's going on. I'm testing this out on FreeBSD 11 with memcached as a backend:
python36-3.6.1_4
libmemcached-1.0.18_6
memcached-1.4.34_2
boto==2.47.0
Collectfast==0.5.2
Django==1.11.2
django-storages==1.5.2
pylibmc==1.5.2
I have collectfast
listed before django.contrib.staticfiles
in INSTALLED_APPS
, a dedicated COLLECTFAST_CACHE
setup using pylibmc, and COLLECTFAST_THREADS
set anywhere from 2 to 20. memcached is started with flags -m 1024 -I 128m
.
Using collectstatic --disable-collectfast
, the static files are copied to the S3 bucket without any problems. When I try it with collectfast enabled, I get all sorts of memcached errors:
The odd thing is that I have other things in Django using memcached, connecting to the same server. In the shell, I can get/set from either the default cache or the dedicated collectfast cache perfectly fine.
I've printed out the calls in etags.py
to see the keys and etags being set, and I can manually set them to memcached in a Python shell without any problems. This only happens when running collectstatic
. When collectstatic
finishes running, Python itself ends in a seg fault.
I switched to using a file-base cache and haven't seen any errors. Any ideas of what's happening?
Re-running CI on 0.6.2 results in errors
https://travis-ci.org/jayvdb/collectfast/builds/495173846
======================================================================
ERROR: collectfast.tests.test_command (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: collectfast.tests.test_command
Traceback (most recent call last):
File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/home/travis/build/jayvdb/collectfast/collectfast/tests/test_command.py", line 7, in <module>
from .utils import test, clean_static_dir, create_static_file, override_setting
File "/home/travis/build/jayvdb/collectfast/collectfast/tests/utils.py", line 10, in <module>
import boto
File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module>
boto.plugin.load_plugins(config)
File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/plugin.py", line 93, in load_plugins
_import_module(file)
File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/plugin.py", line 75, in _import_module
return imp.load_module(name, file, filename, data)
File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/imp.py", line 235, in load_module
return load_source(name, filename, file)
File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "/usr/lib/python2.7/dist-packages/google_compute_engine/boto/compute_auth.py", line 19, in <module>
from google_compute_engine import logger
ModuleNotFoundError: No module named 'google_compute_engine'
======================================================================
ERROR: collectfast.tests.test_etag (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: collectfast.tests.test_etag
Traceback (most recent call last):
File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/opt/python/3.6.3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/home/travis/build/jayvdb/collectfast/collectfast/tests/test_etag.py", line 7, in <module>
from storages.backends.s3boto import S3BotoStorage
File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/storages/backends/s3boto.py", line 25, in <module>
from boto import __version__ as boto_version
File "/home/travis/build/jayvdb/collectfast/.tox/py36-django200/lib/python3.6/site-packages/boto/__init__.py", line 1216, in <module>
boto.plugin.load_plugins(config)
AttributeError: module 'boto' has no attribute 'plugin'
When I try to run it on openSUSE build environment, with latest moto, I get different errors:
======================================================================
ERROR: test_basics (collectfast.tests.utils.test_basics)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/moto/core/models.py", line 74, in wrapper
result = func(*args, **kwargs)
File "collectfast/tests/utils.py", line 33, in wraps
boto.connect_s3().create_bucket(django_settings.AWS_STORAGE_BUCKET_NAME)
File "/usr/lib/python2.7/site-packages/boto/__init__.py", line 141, in connect_s3
return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 194, in __init__
validate_certs=validate_certs, profile_name=profile_name)
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 569, in __init__
host, config, self.provider, self._required_auth_capability())
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 1021, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
The source distribution isnt testable as-is.
Ideally one of the following should work.
python setup.py test
python `which django-admin` test
I see this project uses tox
and runtests.py
.
The files to support that are not included in the sdist.
Is this package compatible with Django 2.1 & 2.2? Anyone using it in production? If so, we should add these versions to tox.ini and travis.yml
Introduce setting for which cache to use and whether or not to use cache at all.
I pip installed the Pypi version, but it still has the last_modified checking in it, and in fact appears to have a bug with that checking mechanism (it skips the upload when local_modified > storage_modified instead of when local_modified < storage_modified , which is working oppositely for me). I notice you've since merged a Pull Request to take out that check all together, with good reason I'm sure.
Anyhow, this is a great library--its taken my collectstatic times down from 10-15 minutes to 2 minutes, which is a huge boost--but its always nice to be able to install a stable version directly from PyPi so updating the PyPi package would be very helpful.
Looks like pypi is current, its just the readme that told me to install the wrong version (0.1.9 instead of 0.1.10)
Hi,
I am getting following error UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 65: ordinal not in range(128)
the full traceback is below:
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/collectfast/management/commands/collectstatic.py", line 70, in handle
super(Command, self).handle(**options)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 193, in handle
collected = self.collect()
File "/usr/local/lib/python2.7/dist-packages/collectfast/management/commands/collectstatic.py", line 63, in collect
Pool(settings.threads).map(self.do_copy_file, self.tasks)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
raise self._value
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 65: ordinal not in range(128)
Looking at traceback I am not sure from which file it is complaining about the error. Please help.
Environment:
- Python 2.7.12
- Ubuntu 16.04.2 LTS
Packages:
- Django==1.10.5
- Collectfast==0.5.2
- boto3==1.4.4
The output of locale
command:
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
Thanks!
Here are my relevant settings:
INSTALLED_APPS += ['storages', 'collectfast', ]
AWS_STORAGE_BUCKET_NAME = os.environ['AWS_STORAGE_BUCKET_NAME']
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
DEFAULT_FILE_STORAGE = 's3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 's3utils.StaticRootS3BotoStorage'
COMPRESS_STORAGE = 's3utils.StaticRootS3BotoStorage'
AWS_PRELOAD_METADATA = True
When I run ./manage.py collectstatic
I can see that it uploads some files. Then I abort the command via CTRL+C and run it immediately again. I can see that it uploads the same files again.
Is there any way for me to debug this?
Boto v3 support was recently added to django-storages (jschneier/django-storages#57), but when using collectfast with boto v3, an error is generated when calling collectstatic: AttributeError: 's3.Bucket' object has no attribute 'lookup'
Running into error during post-processing
django/contrib/staticfiles/storage.py", line 101, in hashed_name:
raise ValueError("The file '%s' could not be found with %r." % (filename, self))
I believe this happens because the underlying collect() command also runs post-process. But because this library overrides collects such that uploads don't happen until the underlying collection function completes. The image has not actually been uploaded yet.
Happy to submit PR to delay the post-process function
This causes the etag comparison to fail and all files to be recopied every time.
See pull request #27
I'm trying to put my static files and uploaded media files in separate folders within my S3 bucket. In order to do that, I'm setting the following:
DEFAULT_FILE_STORAGE = 'myapp.s3utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'myapp.s3utils.StaticRootS3BotoStorage'
s3utils.py is defined as:
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='media')
That causes a problem in get_lookup(path) with the following code:
if cached is False:
self.lookups[path] = self.storage.bucket.lookup(path)
cache.set(cache_key, self.lookups[path])
else:
self.lookups[path] = cached
The problem is that 'path' doesn't exist. The lookup always fails with a 404 not found error. The path needs to be prefixed with the location ('static'). S3BotoStorage has a method '_normalize_name(name)' which does exactly that.
I'm not sure how to fix this generically, but if you change the code to the following, it works just fine:
if cached is False:
self.lookups[path] = self.storage.bucket.lookup(self.storage._normalize_name(path))
cache.set(cache_key, self.lookups[path])
else:
self.lookups[path] = cached
If you don't have boto in your environment (only boto3) has_boto
flag will be set to false in boto.py due to an import error and that will fail the condition check to set preload_metadata to True in the initialization of the mgmt command.
As a consequence collectfast will be slow
This commit adds an import of S3Boto3Storage
with no fallback, which will fail with "django.core.exceptions.ImproperlyConfigured: Could not load Boto3's S3 bindings." if boto3
is not installed.
So long as the intention is to continue to support boto
as well as boto3
, this import should be conditional.
The Exception.message
attribute was removed in python 3 (and deprecated in python 2.6), thus in python 3 a double exception occurs.
At some point, line 38 of collectstatic.py should use e
instead of e.message
.
When i try to do an upload of a file to s3 and get its etag, i get this error
File "/venv3/lib/python3.5/site-packages/collectfast/management/commands/collectstatic.py", line 84, in get_boto3_etag
return self.storage.bucket.Object(path).e_tag
AttributeError: 'Bucket' object has no attribute 'Object'
Django = 1.10.1
python = 3.5
What:
Collectfast==0.6.0
and boto3==1.5.8
don't play well when deploying to Heroku, resulting in
Traceback (most recent call last):
remote: File "manage.py", line 30, in <module>
remote: execute_from_command_line(sys.argv)
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
remote: utility.execute()
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
remote: self.fetch_command(subcommand).run_from_argv(self.argv)
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 206, in fetch_command
remote: klass = load_command_class(app_name, subcommand)
remote: File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 40, in load_command_class
remote: module = import_module('%s.management.commands.%s' % (app_name, name))
remote: File "/app/.heroku/python/lib/python3.6/importlib/__init__.py", line 126, in import_module
remote: return _bootstrap._gcd_import(name[level:], package, level)
remote: File "<frozen importlib._bootstrap>", line 994, in _gcd_import
remote: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
remote: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
remote: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
remote: File "<frozen importlib._bootstrap_external>", line 678, in exec_module
remote: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
remote: File "/app/.heroku/python/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 8, in <module>
remote: from collectfast.etag import should_copy_file
remote: File "/app/.heroku/python/lib/python3.6/site-packages/collectfast/etag.py", line 10, in <module>
remote: from storages.utils import safe_join
remote: ImportError: cannot import name 'safe_join'
remote:
remote: ! Error while running '$ python manage.py collectstatic --noinput'.
remote: See traceback above for details.
I'm also using:
django-storages==1.5.0 # pyup: ==1.5.0
Solution:
Downgrade
Collectfast==0.5.2 # pyup: ==0.5.2
With 1.7 files will get added to S3 but never updated. This may be an django-storages issue though.
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
return self.handle_noargs(**options)
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 163, in handle_noargs
collected = self.collect()
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 113, in collect
handler(path, prefixed_path, storage)
File "/Users/antonagestam/.virtualenvs/fundedbyme/src/collectfast/collectfast/management/commands/collectstatic.py", line 39, in copy_file
source_storage)
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 287, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 219, in delete_file
self.storage.modified_time(prefixed_path)
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/storages/backends/s3boto.py", line 334, in modified_time
return _parse_datestring(entry.last_modified)
File "/Users/antonagestam/.virtualenvs/fundedbyme/lib/python2.7/site-packages/storages/backends/s3boto.py", line 119, in _parse_datestring
raise ValueError("Could not parse date string: " + dstr)
ValueError: Could not parse date string: 2013-05-15T16:27:00.000Z
Hi @antonagestam -- what is the roadmap for getting ready for a 0.1.15 release?
I really like the option of not having to collect statics that haven't changed, I'd love if this became an option to the collectsatic
command itself.
So I was asking myself if you ever discussed that on django-dev
?
Thanks for the project!
Hi ,
I'm trying to move my Django project (which uses collectstatic) from python 2.7 to python 3.6.
When running collectstatic from the command line I get an error (see further). As everything else works fine, I'm a bit lost on what went wrong here.
I did not change my settings.py and all worked fine under python 2.7
Thanks for pointing me in the right direction!
Regards,
Joris
The error:
(platform)baze@baze-mintbox:~/dev/www/projects/ieper/platform > python manage.py collectstatic
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
utility.execute()
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/__init__.py", line 355, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 70, in handle
super(Command, self).handle(**options)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
collected = self.collect()
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 61, in collect
ret = super(Command, self).collect()
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 124, in collect
handler(path, prefixed_path, storage)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 109, in copy_file
self.do_copy_file(args)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 86, in do_copy_file
self.storage, path, prefixed_path, source_storage):
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 93, in should_copy_file
remote_storage, source_storage, path, prefixed_path):
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 83, in has_matching_etag
storage_etag = get_etag(remote_storage, path, prefixed_path)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 58, in get_etag
etag = get_remote_etag(storage, prefixed_path)
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/collectfast/etag.py", line 38, in get_remote_etag
normalized_path = storage._normalize_name(
File "/home/baze/dev/virtualenvs/platform/lib/python3.6/site-packages/django/utils/functional.py", line 239, in inner
return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute '_normalize_name'\
For reference, my pip freeze output:
boto==2.46.1
boto3==1.4.4
botocore==1.5.40
certifi==2017.4.17
chardet==3.0.4
Collectfast==0.5.2
defusedxml==0.5.0
dj-database-url==0.4.2
Django==1.11
django-allauth==0.32.0
django-postgrespool==0.3.1
django-storages==1.5.2
docutils==0.13.1
futures==3.1.1
gunicorn==19.7.1
idna==2.5
jmespath==0.9.2
newrelic==2.86.3.70
oauthlib==2.0.2
psycopg2==2.7.1
python-dateutil==2.6.0
python3-openid==3.1.0
pytz==2017.2
requests==2.17.3
requests-oauthlib==0.8.0
s3transfer==0.1.10
six==1.10.0
SQLAlchemy==1.1.10
urllib3==1.21.1
I have used this library before with one of my S3 projects.
Recently we have moved a project to GCP storage. Collectstatic from a regular server is excruciatingly slow.
I need the benefits of this library for a GCP backend as well.
Can I expect any such changes in the near future?
Hi, thanks for building this promising library. I'm looking forward to using it.
My app uses the default value for STATICFILES_FINDERS
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
When asset are collected into static/, I have these files:
$ ls -alh static
total 4200
drwxr-xr-x 16 ben staff 512B Jan 24 15:49 .
drwxr-xr-x 61 ben staff 1.9K Jan 25 10:59 ..
-rw-r--r-- 1 ben staff 669B Jan 24 15:49 500.html
drwxr-xr-x 6 ben staff 192B Jan 24 15:49 admin
drwxr-xr-x 5 ben staff 160B Jan 24 15:49 debug_toolbar
-rw-r--r-- 1 ben staff 209B Jan 24 15:49 demo_config.yaml
-rw-r--r-- 1 ben staff 671K Jan 24 15:49 demo_deed.pdf
-rw-r--r-- 1 ben staff 101K Jan 24 15:49 demo_statement.pdf
drwxr-xr-x 5 ben staff 160B Jan 24 15:49 django_extensions
drwxr-xr-x 5 ben staff 160B Jan 24 15:49 gis
drwxr-xr-x 13 ben staff 416B Jan 24 15:49 images
-rw-r--r-- 1 ben staff 23B Jan 24 15:49 robots.txt
There are files (500.html, demo_config.yaml) and there are directories (debug_toolbar, images, gis). When I use Collectfast, only the files are uploaded to S3.
$ ./manage.py collectstatic --settings=settings_local --no-input --clear
Copying '/Users/ben/code/minerals/core/static/500.html'
Copying '/Users/ben/code/minerals/core/static/robots.txt'
Copying '/Users/ben/code/minerals/core/static/demo_deed.pdf'
Copying '/Users/ben/code/minerals/core/static/demo_statement.pdf'
4 static files copied.
I expected the files and directories to be uploaded to S3, not just the files. When I use --disable-collectfast
, causing storages.backends.s3boto3.S3Boto3Storage
to be used directly, the files and directories are uploaded to S3.
$ ./manage.py collectstatic --settings=minerals.settings_local --no-input --clear --disable-collectfast
...(lots of copying of a bunch of files)...
226 static files copied.
It's surprising behavior that Collectfast doesn't upload the directories like S3Boto3Storage does when used directly (I think I also tried with S3BotoStorage). Is there a way to get Collectfast to upload the directories?
Here's my config...
STATICFILES_FINDERS = (
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
# I verified that the /var/tmp/collectfast_cache directory is being used as expected
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": "redis://127.0.0.1:6379/2",
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
}
},
'collectfast': {
'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
'LOCATION': '/var/tmp/collectfast_cache',
'TIMEOUT': None,
'OPTIONS': {
'MAX_ENTRIES': 1000,
},
}
}
# Collectfast
STATICFILES_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
AWS_PRELOAD_METADATA = True
COLLECTFAST_THREADS = 20
COLLECTFAST_DEBUG = True
COLLECTFAST_CACHE = 'collectfast'
# I've also setup Boto3 settings correctly (`AWS_STORAGE_BUCKET_NAME`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_QUERYSTRING_EXPIRE`)
Thanks for any help, and thanks for the time you spend on this library. I was surprised at how slow collectstatic was when used with ManifestStaticFilesStorage and S3Boto3Storage, so I was very excited to find this library.
I'm using heroku and django-storage
so say I modified myclass.css, push to heroku, run collectstatics
I would get a myclass_1.css on s3
Please help!
Hi,
Thanks for publishing this project. It is very helpful :-)
I've encountered the following issue:
When using Collectfast in disabled mode (for local development without S3), running ./manage.py collectstatic
breaks even with --disable-collectfast
or COLLECTFAST_ENABLED = False
. The reason seems to be the following:
When using filesystem storage, collectstatic crashes with the following:
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
utility.execute()
File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 216, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 37, in load_command_class
return module.Command()
File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/collectfast/management/commands/collectstatic.py", line 36, in __init__
if self.storage.preload_metadata is not True:
File "/Users/johananl/tmp/collectfast_test/venv/lib/python3.6/site-packages/django/utils/functional.py", line 216, in inner
return func(self._wrapped, *args)
AttributeError: 'StaticFilesStorage' object has no attribute 'preload_metadata'
This happens because StaticFilesStorage
doesn't have an attribute called preload_metadata
(I believe it is S3-specific).
To reproduce this issue, simply run ./manage.py collectstatic
with collectfast
in INSTALLED_APPS
and a default storage configuration.
I believe this can be solved by something similar to the following:
from storages.backends.s3boto3 import S3Boto3Storage
...
if isinstance(self.storage, S3Boto3Storage) and self.storage.preload_metadata is not True:
...
The following is also possible to avoid the import:
if (type(self.storage).__name__ == 'storages.backends.s3boto3.S3Boto3Storage'
and self.storage.preload_metadata is not True):
What do you think?
Thanks,
Johanan
Log errors that are ignored by the try-except in copy_file
(
If AWS_IS_GZIPPED is set to True, then this will check the hash of the un-gzipped item against the etag of the gzipped item, which will obviously not match (though once upon a time it used to, but amazon fixed that).
This means all js and css files on a set-up using gzipping will be copied on every collect.
Current output:
…
Skipping 'dajax/mootools.dajax.core.js'
Skipping 'dajax/jquery.dajax.core.js'
Skipping 'dajax/prototype.dajax.core.js'
Skipping 'dajax/dojo.dajax.core.js'
Copying '/tmp/tmp9FnPsc'
7 static files copied.
Should be less verbose and instead say Skipped 4 files, 7 static files copied.
Getting a traceback with 0.2.0.
[localhost] local: foreman run ./manage.py collectstatic --noinput
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/core/management/base.py", line 415, in handle
return self.handle_noargs(**options)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 178, in handle_noargs
collected = self.collect()
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 60, in collect
ret = super(Command, self).collect()
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 111, in collect
handler(path, prefixed_path, storage)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 135, in copy_file
path, prefixed_path, source_storage)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 294, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/collectfast/management/commands/collectstatic.py", line 141, in delete_file
path, prefixed_path, source_storage)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 224, in delete_file
self.storage.modified_time(prefixed_path)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/storages/backends/s3boto.py", line 334, in modified_time
return _parse_datestring(entry.last_modified)
File "/Users/erik/.virtualenvs/proj/lib/python2.7/site-packages/storages/backends/s3boto.py", line 119, in _parse_datestring
raise ValueError("Could not parse date string: " + dstr)
ValueError: Could not parse date string: 2014-04-11T20:21:13.000Z
Fatal error: local() encountered an error (return code 1) while executing 'foreman run ./manage.py collectstatic --noinput'
Aborting.
I have include license file in the manifest.in file. Could you please make sure it gets into effect in future releases, otherwise we need to add license manually while building it.
I messed up and accidentally had the INSTALLED_APPS order wrong. When I was trying to determine which collectstatic
was working I tried using python manage.py collectstatic --version
. However, even after I fixed the issue it was still reporting my Django version instead of the version of Collectfast.
Yes, it's a really small and pedantic thing, but --version
should be reporting the version of Collectfast.
Setting AWS_PRELOAD_METADATA = True
is not always an option, because it makes some commands in the S3BotoStorage
really slow. But when it's not enabled, the command does not work. So, preload_metadata should be set when it's not set or set to false when running the command.
Relevant:
https://bitbucket.org/david/django-storages/issue/106/preload-breaks-things-like-exist
AGoodId/django-s3-collectstatic#2
For an S3 storage class using the HashedFilesMixin
(e.g. CachedFilesMixin
or ManifestFilesMixin
), a ValueError
will be raised when collectstatic
is run with an empty S3 bucket (or more precisely, a bucket lacking any one of the original static files):
ValueError: The file 'mypath/myfont.eot' could not be found with <myapp.MyStorageClass object at 0x7fa786804978>.
This only occurs when COLLECTFAST_THREADS
is not False
. When the HashedFilesMixin.hashed_name
method uses exists()
to determine whether the file for the cleaned file name exists (on S3), a ValueError
is raised, since it doesn't. i assume this is due to a race condition.
At first i thought that overriding hashed_name
to compute based on file contents of local static media would fix that issue (and be faster?), but for third-party apps, that raises a SuspiciousFileOperation
.
I use the follow code so that I could set the prefix of the location to be the same as the one on the local directory:
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
def path(self, name):
return self.url(name)
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
def path(self, name):
return self.url(name)
~
Prior to 0.5.0, the location
will only affect the result directory on s3, but now it tries to look for the static files while taking this variable into account.
For example. If STATICFILES_LOCATION = "static"
In the past, the search path would have been "/static/js/webpack.js"
Now it is "/static/static/js/webpack.js"
this variable location
should affect where Collectfast is looking for files. This happens not only to the files under /static but for other static files, such as admin related things. There is an extra "/static" in the path.
This was reproduced in 0.5.1 too
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.