miserlou / django-zappa Goto Github PK

Way too much repeated code there, they should all inherit from a ZappaCommand.

Hey all -

First off, great job on all of this! I've been loving Zappa, and can't wait to migrate all of my applications over to using it.

But - I currently have an issue that I would love some clarity on.

I've integrated Boto and S3 static and media hosting, as described here:
https://www.caktusgroup.com/blog/2014/11/10/Using-Amazon-S3-to-store-your-Django-sites-static-and-media-files/

It works quite well, but there is some weirdness when deployed with Zappa.

Specifically, when running locally, all files are uploaded as expected to the S3 servers (I'm just testing through the /admin console).

When running on a Zappa deploy, some uploaded files are mangled (they end up being twice as large as they should be). Generally, I am trying to upload audio files ... and the result is an invalid file.
Text files work fine though ... which leads me to believe it has something to do with escape characters getting added? Since the audio files are non-ASCII in nature.

So ... to sum it up:

Running at localHost:8000, and using /admin ... I can upload FileFields to S3:

• Text files work fine
• Audio files work fine

Running remotely as a Zappa deploy, and using /admin ... I can upload FileFields to S3:

• Text files work fine
• Audio files get horribly destroyed, coming out at about twice the normal file-size

Also, running the same code on an EC2 instance works just fine ... so it must be something with the fact that it is running serverless?

Not sure ... but hoping someone here might have an idea :)

Started in Zappa core, ability to

python manage.py tail beta

And get a tail of the logs going on there.

https://swe19z3zr7.execute-api.us-east-1.amazonaws.com/fadmin/ returns {"message": "Internal server error"}.

The first request to favicon.ico returns a 403 error followed by code 500 for the document.

I'm running into an issue while using python social auth. The problem is when I pass it a token (ex: facebook token). The lambda makes a graph call it never returns. The request always ends up timing out.

Must be the result of the new boto3 stuff?

This code:

        if '~' in settings_file:
            settings_file = settings_file.replace('~', os.path.expanduser('~'))

can be replaced with:

settings_file = os.path.expanduser(settings_file)

Ran "pip install zappa" in a new virtualenv.
I got:
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

and fixed it by:
"sudo apt-get install libffi-dev"

If you look at any travis build you can clearly see that tests should be failing but there are no asserts to test the invoke commands and so they're being called improperly and nothing is detecting the fail.

For developers that work on a team where everyone doesn't have production deployment privileges, loading the settings file from S3 would be helpful. If the user's IAM credentials don't give them access to the S3 file they can't access the settings file. We can already limit the ability to create or update Lambda function code and API Gateway resources via IAM policies.

ZAPPA_SETTINGS = {
'production': {
's3_bucket': 'production-bucket',
'settings_file': 's3://your_bucket:your_folder/production_settings.py',
},
'staging': {
's3_bucket': 'staging-bucket',
'settings_file': '~/Projects/MyApp/settings/staging_settings.py',
},
}

I setup a domain for my AWS Gateway and a default stage so that going to https://sub.domain.com brings up the homepage for the Spheres app. When I try to go to the admin section the app redirects to https://dev/admin/login/?next=/dev/admin/

I have the domain setting setup in the zappa settings:
'domain': 'sub.domain.com',

How do I get the django app to use my custom domain that I've setup and get rid of the /dev/ section of the url?

Hello, first of all sorry if this is in the wrong place.

There is something that got my attetion, about how you need to use python-mysql-connector which is pure python implemented (no C dependencies).
This may not be good if you need high performance on database queries (even more considering a 1 minute timeout as max execution time), and this driver is quite slow compared to the ones with C extensions.

There's mysqlclient connector which does not depend on MySQL Binaries to be installed (but uses C code, and it is one of the fastests connectors) which of course can not be included easily due to the C dependencies. However, I have once deployed python code with C code already compiled on a EC2 machine (there's on AWS Lambda docs the exact ec2 image they use) which worked fine on Lambda, at the cost of having a bigger zip. Have you guys ever considered this as an option if you are looking to include C code?

Add a management command to call zappas rollback function

ie myapp.settings.remote.production and then inspect.getsource

Hey guys!

First of all, thanks for this app, it amazing =D

Im trying to use it with postgres RDS but, the Sphere app does not run. I get always request timed out.
There's any kind of different driver for use postgres with lambda?
Im using this as default engine -> django.db.backends.postgresql_psycopg2

Is that right ?

Thanks in advance! =D

Just created an annoying zip called '..-demo-1457630734.zip'. Let's avoid doing that.

Hello,

Sorry to come with another issue, luckily I am just working on a small test project so switching from python 3.4 to python 2.7.6 was easy enough since I am not doing much that is impacted by switching versions (just testing mostly) though now when I run the deploy, it bundles the files up nicely, creates the lambda app, then create the API Gateway part which seems to complete without issue, but now when Deploying the API Gateway it errors with this: https://gist.github.com/tcross911/369d204d983919a1bbb236831206ef56

I created a project for building leaderboards using redis and it was causing import errors with djang-zappa..

So I have created a django project for trying out django-zappa.
It works fine on my local system but when I hit the api provided by django-zappa, I get following error:

Unable to import module 'handler': No module named django.core.handlers.wsgi

Also the endpoints created by django-zappa in API Gateway look like

Am I supposed to pass some parameters to django-zappa? If yes, how?

Do I need to explicitly create API endpoints in Lambda as I didn't see any in the Lambda console??

Avoid packaging up static files, etc., and whatever else should be excluded. This should arrive shortly in the core lib, just need to add d-z support.

when i tried to use django zappa for configure the django admin panel. session get logged out for every two subsequent clicks. I was unable to reach till adding a uses from django zappa based admin panel. I thought zappa cokkie getting expired.

Hi chaps,

apologies for raising this if it's out of scope.

I've read news about Azure Functions being introduced, and apparently there is less of a cold start issue and Functions are guaranteed to stay hot for at least 5 minutes.

Here is a link to Azure Functions:
https://azure.microsoft.com/en-us/services/functions/

The deployment is successful, however when I tried to access the url, I got {"message":"Forbidden"}. Don't know what happened...

This ended up being way more detailed than I anticipated.
profile.txt

Unfortunately, this is not exposed to boto yet, but maybe we can hack it together before the official SDK is upgraded?

The deploy and update command is not including the package created by the django-admin startproject command. Only the packages that are in the installed apps. This results in a <project_name>.urls import error. No module found zipzip.urls

Hello,

Is there anyone working on postgresql support on RDS ? If not I can try to make it work and submit a pull request.

Thanks

Amine

When I set my ZAPPA_SETTINGS to be something like below

ZAPPA_SETTINGS = {
'dev': {
'aws_region': 'ap-northeast-1',
's3_bucket': 'MY_BUCKET_NAME,
'settings_file': 'PATH_TO_MY_SETTINGS/settings.py'
}
}

the standrad "python manage.py deploy dev" command doesn't work.

However, If I remove the "aws_region" parameter, things magically work. I believe this behaviour is easily reproduciable and might be non-us-region specific.

Would be convenient if had ability to specify which routes API gateway resources/methods were created.

I've had a look myself and I might tackle it soonish. Hopefully.

Thanks for all the hard work on this amazing project!

It is unclear to me how the PYTHONPATH and general environment are setup. I can successfully deploy an application to Lambda, however the deployed Django is unable to find the urls file.

I have, for example, project myproject, with core application myproject (the standard Django setup) which contains urls.py. Thus, the settings files have:

ROOT_URLCONF = 'myproject.urls'

But (in debug mode) I am getting: No module named myproject.urls

The settings I am using work find when running locally. My suspicion is that the PYTHONPATH is not being set so that myproject is available on the path, but this is just a guess really. I noted that the test settings file has BASE_DIR set to an empty string, which strikes me as unusual, but I have no idea if that is related to this. Setting BASE_DIR empty in my own project breaks the update process for me with a botocore function not found error.

Create a command that can be run that will clean up the AWS account after the deploy command (ie restore it to how it was).

Since the IAM role is global, we probably don't want to delete that (by default).

I'm worried that this line might not be unicode safe: https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L105

I'm sorry I'm extremely lazy to create a pull request, but looking at handler.py while trying to help out with an issue at the django forums I noticed some really minor optimizations that you can do (they are really minor and won't change any timing significantly, but it is always good to do things in the best way as possible, every milisecond counts!)

Debug and logger calls: Are you using AWS logging? If I'm not wrong that goes to cloud watch which can be pretty slow, I would really avoid any logging when debug is false and do not test any timings with code that writes to those logs.

Line 61, def start(a,b)... Defining a function over and over is quite expensive, and nested functions are a bit more expensive on python also. I would have that one defined at module level only once.

Line 87: if response.status_code in [400, 401, 403, 404, 500]: You are defining this list over and over, and also the IN operator on a list is always O(n) worst case, I would define this list as a global variable and even better make it a set or dictionary which will be O(1) most of the time (although for such a small list, the difference is really small, on a 10000000 loop it is about 25% faster on python2.7)

Overall, try to use either format or % operator for string concatenation, it is way more efficient and readable than +

I couldn't find how does the common_log work, but you are calling it every time even with no debug, which might be giving you some unnecessary slowdowns.

Would be very cool to showcase some Zappa-powered instances of

• Pinax
• Django-CMS
• Wagtail
• Mezzanine
• django-machina

and would also help us discover the bugs/quirks necessary for deploying those.

..any others?

Raises a warning if the Middleware isn't in the settings, even if it would be loaded safely by calling from "from .base import *", etc

When I try to deploy I get this error after the zip file is uploaded and after Creating API Gateway routes.. is printed to the console:

botocore.exceptions.ClientError: An error occurred (NotFoundException) when calling the PutIntegrationResponse operation: No integration defined for method

Any ideas what I might be doing wrong?

Hello,

I have tried multiple times to run the deploy from various environments, and each time I run into this set of errors: https://gist.github.com/anonymous/0b078b15465bb3ab104e3ed9f631acfb

I am running Django 1.9.5 with Python 3.4.4

There appears to be a problem with the certification authority of the SSL certificate for https://zappa.gun.io/.

It's annoying to upload a zip, only to realize that

- You are over the limit of APIs
- There is already an API with this name

There should be a pre_deploy function to prevent those annoyances.

Generally need a better understanding and documentation about static/media/admin file handling with D-Z. Should come soon.

I tried deploying, and keep getting this error when deploying Sphere.

~/.aws$ cat credentials
[default]
aws_access_key_id = [censored]
aws_secret_access_key = [censored]

The only file I really needed to change was spheres/production_settings.py

Followed the video, and seems like I'm just getting a 500 Internal Server Error?
https://l0723k4q1j.execute-api.us-east-1.amazonaws.com/production

Any relative imports in settings files do not work as it is being copied to a different location.

It was pointed out during the Q & A that the Django Middleware could be replaced with a WSGI Middleware that would live in Zappa core. This has a number of obvious benefits, such as more shared code, less client-specific code, and less client-specific integration.

It looks like the repo needs a singular static requirements.txt file - I'm quite used to having at least 3, one for common stuff, one for dev and one for prod (and the latter 2 include the former).

I had a quick skim of the code and it looks like it's looking for the lines of requirements.txt in setup.py -- looks like there's a package (requirements-parser) that can slot in here, making the requirements an arg somewhere:

In [1]: import requirements
In [2]: fp = open('requirements/dev.txt')
In [3]: list(requirements.parse(fp))
Out[3]:
[<Requirement: "Pillow==2.8.1">,
 <Requirement: "boto==2.38.0">,
 <Requirement: "hashids==1.1.0">,`
...

This question gets asked a lot and should be included in the docs

So I deployed my app in Lambda, with the URL like this https://16xf96vt2l.execute-api.eu-central-1.amazonaws.com/production/.

How do I remove /production/ on it?

The implicit status code handling for redirects will moste likely misbehave if any other than a 200 response code provides a location.

A good example would be a 201 CREATED, should include a location header.

I guess it makes more sense to filter for 3xx status codes rather than NOT 200.

This should probably come after #11, but it'd be great to be able to profile a request so that we can start shaving some time off each response. I bet we can get it below 100ms.

512 is probably too high for a default, I bet we can drop that by half at least.