logandk / serverless-wsgi Goto Github PK
View Code? Open in Web Editor NEWServerless plugin to deploy WSGI applications (Flask/Django/Pyramid etc.) and bundle Python packages
License: MIT License
Serverless plugin to deploy WSGI applications (Flask/Django/Pyramid etc.) and bundle Python packages
License: MIT License
I was trying the new django management commands.
My serverless.yml contains environmental variables for different stages and in order to run the wsgi server locally I do: sls wsgi serve -s local
.
I would also like to execute the django management commands locally such as with sls wsgi manage -c "check --list-tags" -s local
.
It seems to not work out as I get the following error:
Function not found: arn:aws:lambda:eu-central-1:xxxxxx:function:cookbook-local-app
Is this just not possible or am I missing something?
Currently, the "sls wsgi serve" command has an error when calling wsgi.
It will return: "Unable to import werkzeug (run: pip install werkzeug)"
even though the werkzeug requirement is in the requirements.txt file of the wsgi package.
The steps to directly reproduce on a fresh Ubuntu 16.04 VirtualBox Install can be found here:
https://serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/
Just for kicks:
-I installed werkzeug locally and sudo cp'd it into the serverless-wsgi main directory and there was another requirement that needed to be installed (boto3, botocore, click, etc). I did this for a couple of iterations before coming here.
-I pip installed everything needed for the ^^ dependency tree failure in a virtualenv and pip freezed the venv into the serverless-wsgi requirements.txt , also no avail.
This stems from issues I was having here in handling multipart/form-data with serverless-wsgi.
I've used the serverless-apigw-binary
plugin as recommended by the File Uploads section of the Readme. I've isolated my problem to this line in wsgi.py
. Pasting some code in here for context:
75 body = event[u'body'] or ''
76 if event.get('isBase64Encoded', False):
77 body = base64.b64decode(body)
78 body = to_bytes(wsgi_encoding_dance(body))
The body is Base 64 encoded, so it goes through the decoding in line 77.
After the body is base64 encoded but before it goes through the wsgi_encoding_dance
and to_bytes
functions, the body is a bytes object that is the proper image. If I comment out line 78, it works fine.
I added some logging around line 78, and the bytes object is changed when decoding from bytes to str in the wsgi_encoding_dance()
and then the reconversion to bytes in to_bytes
. I logged the length before and after the line, and the body went from a length of 145647 to 218875.
I don't know enough about encoding & decoding to fix this myself, so looking for advice. Would it be possible to skip the WSGI encoding dance & to_bytes conversion if the body is already a bytes
object? If not, is there some other proxy we can look at to see whether the body needs to go through that conversion?
Let me know if you need any more details ๐
When I run serverless wsgi serve
I'm having problems with the requests taking a long time (10seconds - 1min) to return a response. Sometimes the response will not return at all. To avoid hitting a database I'm sending GET request to a route that immediately returns 'hello world' 200.
So far the only predictable behaviour I can find is that making a request using jquery will always cause a long delay before returning. If I try to request the same URL in the browser tab (whilst the jquery request is still waiting) the tab also waits a long time for for the request to return.
However, once the jquery request returns, calling the URL in a browser tab then returns immediately. Refreshing the tab gives a response immediately.
Sometimes, if jquery request is pending, calling the URL in a browser tab WILL return immediately. When this happens, this seem to also make the jquery request return a response.
Jquery code below:
$( document ).ready(function() {
$.get("http://localhost:5000/test", function(data, status){
alert("Data: " + data + "\nStatus: " + status);
});
});
I've tried restarting the server, browsers and my machine, but the problem still persists.
I've tried multiple browsers (safari, chrome, firefox) and all have the same issue.
I've tried making a javascript request using axios instead of jquery but still the requests are returning slowly.
Does anyone have any idea how I might solve this or any additional debugging I can do?
I'm using:
Due to a quirk in API Gateway, multiple set-cookie's will be dropped to a single one. Mangling the case of the set-cookie header is sufficient to pass them through though.
This pluging should do the case mangling on the way out, perhaps via middleware as Zappa does, to remedy this. Here's what my PR to Zappa to do this looked like:
Miserlou/Zappa#636
The final code is fairly simple, perhaps it can be more directly incorporated into the WSGI handler:
https://github.com/Miserlou/Zappa/blob/master/zappa/middleware.py
I'm using serverless to deploy a Django (REST Framework) API as an AWS Lambda function, and I'm also going to be deploying other Lambda functions that are not WSGI apps. I have this all in one repo right now, with serverless.yml
alongside api
, where api
is the Django project directory.
Thus, I set custom.wsgi.app: api.api.wsgi.application
. This allows the WSGI handler to find my app just fine, but when the app runs, it appears that the working directory is still the root of my repo, rather than the root of the Django app, because I get an import error on api.settings
. I get the same error message when deploying the endpoint to AWS and testing there.
Sorry if this is misfiled; I'm not sure if this is a question about Django, serverless, or serverless-wsgi, as I'm new to all of them. :) Is there a way to make sure the function that runs my Django app from a working directory other than where serverless.yml
lives? Is that up to serverless or serverless-wsgi, or Django?
When I have unicode characters in my paylaod the CONTENT_LENGTH seems to be different than the actual wsgi.input length that I'm sending to my flask app
input: '{"test":"รฉ"}'
len: 12 (should be 13, afaik)
Using django-rest-framework-json-api, all of the responses are unnecessarily base64 encoded. The fix is simple. In wsgi.py
, add application/vnd.api+json
to TEXT_MIME_TYPES
.
Thank you.
Use case: Django on AWS
Any attempt to access the POST body with serverless-wsgi deployed on AWS results in Django trying to concatenate a bytes
and a str
- for example, the Django admin page:
Environment:
Request Method: POST
Request URL: https://<redacted>.execute-api.us-east-1.amazonaws.com/dev/admin/login/?next=%2Fdev%2Fadmin%2F
Django Version: 1.11.4
Python Version: 3.6.1
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'rest_framework.authtoken']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "/var/task/django/core/handlers/exception.py" in inner
41. response = get_response(request)
File "/var/task/django/core/handlers/base.py" in _get_response
178. response = middleware_method(request, callback, callback_args, callback_kwargs)
File "/var/task/django/middleware/csrf.py" in process_view
298. request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')
File "/var/task/django/core/handlers/wsgi.py" in _get_post
132. self._load_post_and_files()
File "/var/task/django/http/request.py" in _load_post_and_files
311. self._post, self._files = QueryDict(self.body, encoding=self._encoding), MultiValueDict()
File "/var/task/django/http/request.py" in body
272. self._body = self.read()
File "/var/task/django/http/request.py" in read
331. return self._stream.read(*args, **kwargs)
File "/var/task/django/core/handlers/wsgi.py" in read
50. result = self.buffer + read_data
Exception Type: TypeError at /admin/login/
Exception Value: can't concat bytes to str
The str
comes from serverless-wsgi, which uses a StringIO
to pass wsgi.input
to the wsgi app. This should be fixed to use a BytesIO
in Python 3, as Zappa has done:
Posting a PR shortly, after I do some testing.
I have service which has a WSGI application and some regular lambda functions.
The non-WSGI code api.py
is this:
# https://github.com/logandk/serverless-wsgi/#automatic-requirement-packaging
import os
import sys
root = os.path.abspath(os.path.join(os.path.dirname(__file__)))
sys.path.insert(0, os.path.join(root, '.requirements'))
import xmltodict
def handler(event, context):
return xmltodict.unparse({"foo": "BAR"})
serverless.yml:
functions:
web:
handler: wsgi.handler
events:
- http: ANY /
- http: ANY {proxy+}
api:
handler: api.handler
custom:
wsgi:
app: app.app
requirements.txt
xmltodict
After a full deploy, the non-WSGI code is fine.
$ sls deploy -v
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (2.06 MB)...
...
$ sls invoke -f api -l
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<foo>BAR</foo>"
However, after redeploying just the function, it fails.
$ sls deploy function -f api
Serverless: Packaging function: api...
Serverless: Excluding development dependencies...
Serverless: Uploading function: api (2.02 KB)...
Serverless: Successfully deployed function: api
$ sls invoke -f api -l
{
"errorMessage": "Unable to import module 'api'"
}
--------------------------------------------------------------------
START RequestId: 94f3188a-9239-11e7-8d0e-01bfd0695980 Version: $LATEST
Unable to import module 'api': No module named 'xmltodict'
Notice how the sls deploy function
only uploads 2.02KB.
Aside: having a service which has both WSGI and non-WSGI components is very useful to me. Use cases:
(so it would be nice to have first-class support for such functions, so that they don't need to add .requirements
to the path themselves)
using 'wsgi serve' definitely helps speed up development, but i'm looking into running the project directly with a line by line debguger (pydev). it appears i'm having an issue due to environment variables not being read in by serverless as they are when using 'wsgi serve'.
not a big problem to overcome, but i thought i'd ask if there are any recommendations for debugging with breakpoints.
I am running under ubuntu 16.04. This has python3
installed but no python
. python(2) is an optional package, and the system doesn't install it by default. I'd like to keep my system python2-free.
I am trying the sample flask app, and only slightly modified serverless.yml
:
service: my-flask
provider:
name: aws
runtime: python3.6
stage: dev
region: eu-west-1
plugins:
- serverless-wsgi
functions:
api:
handler: wsgi.handler
events:
- http: ANY /
- http: ANY {proxy+}
custom:
wsgi:
app: api.app
When I try to deploy I get the following error:
$ serverless deploy -v
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Error --------------------------------------------------
spawnSync python ENOENT
There are two places in index.js which have child_process.spawnSync('python', ...
so I changed them both to python3
Now I get a different error:
Error --------------------------------------------------
Unable to load virtualenv, please install
Fixed by apt-get install python3-virtualenv
But now it fails, and the error message is completely blank :-(
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Error --------------------------------------------------
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Setting SLS_DEBUG=1 I get:
Error
at module.exports.logError.errorHandlingError (/usr/lib/node_modules/serverless/lib/classes/Error.js:94:11)
at initializeErrorReporter.then.catch.e (/usr/lib/node_modules/serverless/bin/serverless:42:3)
at runCallback (timers.js:672:20)
at tryOnImmediate (timers.js:645:5)
at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
at __dirname (/usr/lib/node_modules/serverless/bin/serverless:40:9)
at Object.<anonymous> (/usr/lib/node_modules/serverless/bin/serverless:43:4)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:389:7)
at startup (bootstrap_node.js:149:9)
at bootstrap_node.js:504:3
Still no good. That seems like a bug in its own right - error messages shouldn't be discarded!
So I bring strace to bear:
strace -s 128 -f serverless deploy -v
...
and now I see the error:
[pid 5633] read(21, "The executable python2 (from --python=python2) does not exist\n", 65536) = 62
Using virtualenv in Ubuntu requires virtualenv -p python3
Modified requirements.py
:
sys.argv = ['', venv_dir, '--quiet', '-p', 'python3']
After this, deployment works, phew.
I'm not sure of the right clean fix. Maybe there should be a setting likecustom.wsgi.python: python3
which overrides both the binary and adds the -p
option to virtualenv?
I do not know if this is already implemented but I have not found it in the docs. My apologies in advance If it can be done easier.
I find myself including a serverless env variable reading an opts
argument to check if my code is running locally to disable some functionalities (logs, remote module loading...)
environment:
DEBUG_SERVER: ${opt:debugServer, 'false'}
It would be nice if this were built in transparently from serverless via (werkzeug or the wsgi handler), I have checked the environment of the lambda function and I haven't seen anything like this.
I can help with this if you agree and confirm the proper design.
PS: Thanks for your amazing work, you saved us a lot of time sir.
serverless-wsgi currently sets the REMOTE_ADDR
variable like this:
'REMOTE_ADDR':
headers.get(u'X-Forwarded-For', '').split(', ')[0],
But that's insecure, since anyone can submit a request with their own X-Forwarded-For:
header, containing one or more IPs. API gateway will append to the end of this, so the header becomes:
X-Forwarded-For: <spoofed-address(es)>, <real-source>, <api-gateway>
and REMOTE_ADDR
will be set to the first spoofed address.
What should be returned is <real-source>
, which you can get via:
'REMOTE_ADDR':
headers.get(u'X-Forwarded-For', '').split(', ')[-2],
Of course that will fail if for some reason the header is missing or broken, so safer would be:
ips = headers.get(u'X-Forwarded-For', '').replace(' ','').split(',')
if len(ips) == 1:
remote_addr = ips[0]
else:
remote_addr = ips[-2]
...
'REMOTE_ADDR':
remote_addr
Another option is to use event["requestContext"]["identity"]["sourceIp"]
, although I've not tested whether that is spoofable or not.
I'm trying to use environment variables to access CF stack outputs and the following works for me in Lambda:
environment:
VAR1:
Fn::ImportValue: "UserPool::Id"
# OR
VAR2:
Ref: UserPool
But it won't work for sls wsgi serve
:
>>> os.environ['VAR1']
[object Object]
>>> os.environ['VAR2']
[object Object]
I don't know much about how serverless plugins work but my best guess is that it comes from here:
https://github.com/logandk/serverless-wsgi/blob/master/index.js#L163
Is this a serverless problem? or would it be possible for serverless-wsgi to store the resolved Fn::ImportValues on sls deploy
and reproduce them later? or pull from cloudformation?
I'm not certain of the best way to debug this problem, but my WSGI app (Falcon, in this case) sets multiple Set-Cookie
headers on certain responses and the cookies are correctly set when running locally.
However, when running the same code after deploying to Lambda (using serverless-wsgi), my application works entirely as expected with the notable exception that all responses only ever contain a single set-cookie
header with a single cookie value in them.
This problem is evident in Chrome and Safari (on Mac).
We've run into a similar situation as described in #55 .
We request to please add 'application/problem+json' to the TEXT_MIME_TYPES list in https://github.com/logandk/serverless-wsgi/blob/1.4.7/wsgi.py#L21 .
To add some context, we are using serverless-wsgi with zalando/connexion and the connexion framework often returns mimetype 'appliaction/problem+json' as seen here: https://github.com/zalando/connexion/blob/1.5.2/connexion/problem.py#L39
This appears to trip up serverless-wsgi into incorrectly encoding the response body in base64. Adding the new entry to the TEXT_MIME_TYPES list appears to fix the issue.
Thank you for your time.
Using Flask, I cannot see a way to access the raw 'event' from within a function, although the 'context' is available:
'context':
context,
So for example, I can't get to low-level things like event["requestContext"]["identity"]["sourceIp"]
Have I overlooked something? Otherwise, could it be added like this:
'event':
event,
Hi ,
Following by the blog post, https://serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/#converting-an-existing-flask-application.
Just wondering how can I get the lambda event and context ? can I just print event
as below?
tried but it did not work
I would appreciate it if anyone can help.
# app.py
import os
import boto3
from flask import Flask, jsonify, request
app = Flask(__name__)
event = os.environ['event']
print event
if IS_OFFLINE:
client = boto3.client(
'dynamodb',
region_name='localhost',
endpoint_url='http://localhost:8000'
)
else:
client = boto3.client('dynamodb')
@app.route("/")
def hello():
return jsonify(os.environ)
... rest of application code ...
https://github.com/logandk/serverless-wsgi/blob/master/wsgi.py#L144
This needs to be changed like was done on the Zappa framework:
https://github.com/Miserlou/Zappa/pull/938/files#diff-b4b0b6c0ef632d8dc89e72cca21a5cdeR448
Otherwise, the hidden wrapper function that Lambda uses fails to json.dumps() the object.
Please resolve ASAP. We're blocked using Python 3.6. I'm surprised no one else has complained about this yet.
I started having this issue today when deploying python lambda functions, and it's hard to say what caused this, since I didn't update any relevant configuration. The cloudwatch logs also only show:
Unable to import module 'wsgi': No module named 'werkzeug'
with no other backtrace, and this seems to happen before my application code is evaluated. I also tried to revert to the latest stable release, and it also doesn't work.
Did something in AWS change which caused this error?
Custom authorizers have a serious flaw in that they can not return headers. This means that the CORS header can not be set for 401's.
To work around this I will "allow" every request, but include the authorization status and reason(missing token, invalid token, expired, etc) in the authorizer context. However serverless-wsgi does not currently extract those keys.
I would like to setup a way to extract these authorizer keys...
Any chance this could be released as a standalone python module. Looks like it could be really useful outside the serverless framework.
Suddenly getting this error again on v1.6.1
Rolling back to 1.6.0 fixes the issue
Serverless: WARNING: Plugin ServerlessWSGI uses deprecated hook before:deploy:createDeploymentArtifacts,
use package:createDeploymentArtifacts hook instead
serverless-wsgi version 1.3.0, serverless 1.23.0
When running in python3.7 virtualenv this throws:
spawnSync python3.6 ENOENT
If an app has gained a top-level __pycache__
directory (e.g. because you ran the app locally to test it) then this causes sls deploy
to fail:
$ sls deploy
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Linking required Python packages...
Serverless Error ---------------------------------------
Unable to link dependency '__pycache__' because a file by the same name exists in this service
Workaround is to rm -rf __pycache__
first.
(Aside: I wonder if the requirements bundle should also exclude *.pyc
files, so they are rebuilt server-side when the application starts up)
package has requirement - werkzeug, but this is not written
in setup.py
Add command to automatically determine routes from (Flask) app and update serverless.yml
.
Or... is there difference between the two?
I'm not sure if/how I'm getting this behavior but unless I've added this mime type (application/javascript) here:
https://github.com/logandk/serverless-wsgi/blob/master/wsgi.py#L16
All of my javascript files loading in are base64 encoded and break rendering. I've adding this mime type and everything is fixed. This seems like a bug unless I'm overlooking something in configuration.
I following the tutorial here: https://serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/:
It was working wonderfully for me, but I've come back to it a few weeks later (all in git, so no changes) and I'm now getting the following error:
Traceback (most recent call last):
File "/Users/aloughran/Code/node_modules/serverless-wsgi/serve.py", line 50, in <module>
serve(*sys.argv[1:])
File "/Users/aloughran/Code/node_modules/serverless-wsgi/serve.py", line 42, in serve
use_evalex=True)
File "/Library/Python/2.7/site-packages/werkzeug/serving.py", line 720, in run_simple
s.bind((hostname, port))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known.
The problem is more than likely an issue with my configuration, but I'm lost on working out where the nodename/servname should be passed by the wsgi config, or what could be tripping me up.
Thanks in advance.
Spent a good few hours trying to work out why an image/svg+xml
MIME was returning garbage:
Should return something like:
<?xml version="1.0"?><svg width="1125" height="432" xmlns=...
But was returning:
PD94bWwgdmVyc2lvbj0iMS4wIj8 ....
I then read the following in the docs:
Text MIME types
By default, all MIME types starting with text/ and the following whitelist are sent through API Gateway in plain text. All other MIME types will have their response body base64 encoded (and the isBase64Encoded API Gateway flag set) in order to be delivered by API Gateway as binary data.
It then dawned on me that image/svg+xml
needs to be returned as text, not binary (like most images).
Fixed by adding the following to serverless.yml:
textMimeTypes:
- image/svg+xml
Whilst this may be an edge case, it may be helpful to add image/svg+xml
to the whitelist as I went round and round in circles in my attempt to solve this.
Since SVG images are mainly used in use cases where you need to real-time build an image in code (instead of serving one directly from s3) its seems like a good fit for the whitelist in my opinion.
My requirements.txt is frozen, and I don't want the pip install to pull some python 2 dependencies (my lambda runtime is python3).
Concrete example:
(make new virtualenv)
$ pip-install python-jose
$ pip freeze > requirements.txt
$ cat requirements.txt
ecdsa==0.13
future==0.17.1 <<<<<<<<< remove this line (python2 only, size is 1.5 megs)
pyasn1==0.4.4
python-jose==3.0.1
rsa==4.0
six==1.11.0
This is a matter of keeping the lambda function zip file size down. But if I do this currently, the serverless-wsgi plugin re-pulls the (transitive) future
dependency. I don't want that.
A quick test shows that my code would still work without the future
dependency.
$ pip uninstall future
Uninstalling future-0.17.1:
(...)
Proceed (y/n)? y
Successfully uninstalled future-0.17.1
$ python
Python 3.6.6 (default, Aug 26 2018, 13:00:14)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from jose import jwt
>>> token = jwt.encode({'key': 'value'}, 'secret', algorithm='HS256')
>>> print(token)
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJrZXkiOiJ2YWx1ZSJ9.JPIDicqvQ6GAh14yE2yZ3wnZQ0LiLNTTRDtJgLZcn98
Could a capability be added to pass --no-deps
to the pip install command-line run in requirements.py? Maybe a way to pass pip command-line arguments through the serverless.yml
's custom:
section, or some environment variable?
I'll try the python-requirements plugin to see what happens there.
Hi @logandk, community members,
Thanks for creating and open sourcing this amazing package :)
While working with open whisk, I had this little issue :
$ sls deploy
Serverless: Packaging Python WSGI handler...
Serverless: Packaging required Python packages...
Serverless: Packaging service...
Serverless: Compiling Functions...
Serverless Error ---------------------------------------
Function handler (wsgi.py) does not exist.
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Forums: forum.serverless.com
Chat: gitter.im/serverless/serverless
Your Environment Information -----------------------------
OS: darwin
Node Version: 6.2.2
Serverless Version: 1.15.1
It said that wsgi.py does not exist, but I verified that the file was present in the root directory when I ran the command.
Can you please help with this ?
Changes from #33 don't appear to work with ubuntu 16.04. Reproducing case:
serverless.yml
service: my-service
provider:
name: aws
runtime: python3.6
stage: dev
region: eu-west-1
package:
exclude:
- "**"
include:
- handler.py
functions:
hello:
handler: handler.hello
events:
- http:
path: /foo
method: get
plugins:
- serverless-wsgi
custom:
wsgi:
app: handler.hello
pythonBin: python3
handler.py
import xmltodict
def hello(event, context):
body = {
"message": "Hello lambda world",
"input": event
}
response = {
"statusCode": 200,
"body": xmltodict.unparse({"result": body}),
}
return response
requirements.txt
xmltodict
Results:
{"message": "Internal server error"}
.Unable to import module 'handler': No module named 'xmltodict'
.serverless/my-service.zip
is huge (20.52MB) and contains a load of unrelated junk in a .venv
top-level directory.$ unzip -v .serverless/my-service.zip
Archive: .serverless/my-service.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
241 Defl:N 152 37% 1980-01-01 00:00 4c3917f2 handler.py
6091 Defl:N 2497 59% 1980-01-01 00:00 04039966 wsgi.py
13 Defl:N 15 -15% 1980-01-01 00:00 a6e25e52 .wsgi_app
2108 Defl:N 833 61% 1980-01-01 00:00 84ee13bb .venv/bin/activate
1137 Defl:N 537 53% 1980-01-01 00:00 078145a3 .venv/bin/activate_this.py
1050 Defl:N 532 49% 1980-01-01 00:00 6824902d .venv/bin/activate.csh
2248 Defl:N 900 60% 1980-01-01 00:00 82d1587f .venv/bin/activate.fish
278 Defl:N 206 26% 1980-01-01 00:00 514e61e4 .venv/bin/easy_install
278 Defl:N 206 26% 1980-01-01 00:00 514e61e4 .venv/bin/easy_install-2.7
250 Defl:N 188 25% 1980-01-01 00:00 e0fcf0e5 .venv/bin/pip
250 Defl:N 188 25% 1980-01-01 00:00 e0fcf0e5 .venv/bin/pip2
250 Defl:N 188 25% 1980-01-01 00:00 e0fcf0e5 .venv/bin/pip2.7
5429482 Defl:N 1653333 70% 1980-01-01 00:00 46c4a7b2 .venv/bin/python
2367 Defl:N 905 62% 1980-01-01 00:00 9b7d9935 .venv/bin/python-config
5429482 Defl:N 1653333 70% 1980-01-01 00:00 46c4a7b2 .venv/bin/python2
5429482 Defl:N 1653333 70% 1980-01-01 00:00 46c4a7b2 .venv/bin/python2.7
257 Defl:N 193 25% 1980-01-01 00:00 8fcca9bf .venv/bin/wheel
18619 Defl:N 5186 72% 1980-01-01 00:00 b0f6c3cf .venv/lib/python2.7/_abcoll.py
30978 Defl:N 8802 72% 1980-01-01 00:00 5f62d05f .venv/lib/python2.7/_abcoll.pyc
5911 Defl:N 1619 73% 1980-01-01 00:00 0d0944be .venv/lib/python2.7/_weakrefset.py
12100 Defl:N 2989 75% 1980-01-01 00:00 bbd04380 .venv/lib/python2.7/_weakrefset.pyc
7145 Defl:N 2393 67% 1980-01-01 00:00 c9f117b3 .venv/lib/python2.7/abc.py
6780 Defl:N 2784 59% 1980-01-01 00:00 8d2304aa .venv/lib/python2.7/abc.pyc
...
151008 Defl:N 72686 52% 1980-01-01 00:00 9d6eb4c8 .venv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl
118619 Defl:N 56392 53% 1980-01-01 00:00 169d61c0 .venv/share/python-wheels/wheel-0.29.0-py2.py3-none-any.whl
-------- ------- --- -------
67493747 21060069 69% 2104 files
Note that the .venv
subdirectory appears to include a bunch of python2 stuff, even though I have specified pythonBin: python3
.
And it doesn't include the package from requirements.txt
$ unzip -v .serverless/my-service.zip | grep xmltodict
$
I am using the serverless-wsgi with a Django Rest Framework project.
Using the browsable API functionality of DRF, when I run the local server using sls wsgi serve -p 8000
, I get loads of 404 errors for the rest framework's static files. When I run the local server using ./manage.py runserver
, I get no 404 errors.
EDIT: Not a big deal, since I am not going to use the browsable api functionality in production anyways, but it's great for testing locally
See the contrasting screenshots below
After testing the plugin with the project very much similar to the one in example, in which local tests work correctly, after deployment to AWS I get the following message on call:
{
"errorMessage": "Unable to import module 'wsgi'"
}
Detailed log:
START RequestId: 6b36272e-920d-11e6-9aed-0d5b9d25097b Version: $LATEST
Unable to import module 'wsgi': No module named datastructures
END RequestId: 6b36272e-920d-11e6-9aed-0d5b9d25097b
REPORT RequestId: 6b36272e-920d-11e6-9aed-0d5b9d25097b Duration: 0.41 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 15 MB
It seems that the requirements are not correctly visible to WSGI. Everything in the deployment package seems to be in the right place.
Thanks for the help.
I would very much like to use the CodeSha256 attribute to check whether multiple stages are running identical code.
Unfortunately, every time I do sls deploy
, even to the same stage, the generated zip file has a slightly different content and therefore different hash.
This turns out to be due to changes in the *.dist-info
from pip packages added by serverless-wsgi
- details below. Stripping this out makes the sha256 stable.
Since the dist-info data is not required for successful running of the library, please could serverless-wsgi
filter it out by default?
Demonstration:
$ sls deploy && cp .serverless/foo.zip a.zip
...
$ sls deploy && cp .serverless/foo.zip b.zip
...
$ ls -l a.zip b.zip
-rw-rw-r-- 1 ubuntu ubuntu 8011038 Jan 16 10:35 a.zip
-rw-rw-r-- 1 ubuntu ubuntu 8011006 Jan 16 10:36 b.zip
$ unzip -v a.zip >a; unzip -v b.zip >b; diff -u a b
--- a 2018-01-16 10:37:11.733316280 +0000
+++ b 2018-01-16 10:37:11.753317413 +0000
@@ -1,4 +1,4 @@
-Archive: a.zip
+Archive: b.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
9264 Defl:N 3823 59% 1980-01-01 00:00 d15c6bba api.py
@@ -200,7 +200,7 @@
1534 Defl:N 819 47% 1980-01-01 00:00 66908bfa Werkzeug-0.14.1.dist-info/LICENSE.txt
3872 Defl:N 1672 57% 1980-01-01 00:00 5966375f Werkzeug-0.14.1.dist-info/METADATA
1452 Defl:N 659 55% 1980-01-01 00:00 2f6f6a96 Werkzeug-0.14.1.dist-info/metadata.json
- 6775 Defl:N 3018 56% 1980-01-01 00:00 019bf153 Werkzeug-0.14.1.dist-info/RECORD
+ 6775 Defl:N 3003 56% 1980-01-01 00:00 4ec41c26 Werkzeug-0.14.1.dist-info/RECORD
9 Defl:N 11 -22% 1980-01-01 00:00 d4cd6b6c Werkzeug-0.14.1.dist-info/top_level.txt
110 Defl:N 95 14% 1980-01-01 00:00 aa646eec Werkzeug-0.14.1.dist-info/WHEEL
63 Defl:N 61 3% 1980-01-01 00:00 8eaf4047 certifi/__init__.py
@@ -213,7 +213,7 @@
4 Defl:N 6 -50% 1980-01-01 00:00 c2971fc7 certifi-2017.11.5.dist-info/INSTALLER
2546 Defl:N 1145 55% 1980-01-01 00:00 c2b36ca6 certifi-2017.11.5.dist-info/METADATA
913 Defl:N 410 55% 1980-01-01 00:00 8a2edf06 certifi-2017.11.5.dist-info/metadata.json
- 1201 Defl:N 688 43% 1980-01-01 00:00 61dc9d14 certifi-2017.11.5.dist-info/RECORD
+ 1201 Defl:N 688 43% 1980-01-01 00:00 4755d1e9 certifi-2017.11.5.dist-info/RECORD
8 Defl:N 10 -25% 1980-01-01 00:00 f6eec75d certifi-2017.11.5.dist-info/top_level.txt
113 Defl:N 98 13% 1980-01-01 00:00 7b96df94 certifi-2017.11.5.dist-info/WHEEL
1731 Defl:N 883 49% 1980-01-01 00:00 21a13c64 certifi-2017.7.27.1.dist-info/DESCRIPTION.rst
@@ -269,7 +269,7 @@
4 Defl:N 6 -50% 1980-01-01 00:00 c2971fc7 chardet-3.0.4.dist-info/INSTALLER
3239 Defl:N 1403 57% 1980-01-01 00:00 c8754a44 chardet-3.0.4.dist-info/METADATA
1375 Defl:N 614 55% 1980-01-01 00:00 e8bec562 chardet-3.0.4.dist-info/metadata.json
- 6167 Defl:N 2631 57% 1980-01-01 00:00 390df0c2 chardet-3.0.4.dist-info/RECORD
+ 6167 Defl:N 2623 58% 1980-01-01 00:00 53b58155 chardet-3.0.4.dist-info/RECORD
8 Defl:N 10 -25% 1980-01-01 00:00 927b2990 chardet-3.0.4.dist-info/top_level.txt
110 Defl:N 95 14% 1980-01-01 00:00 1382dbab chardet-3.0.4.dist-info/WHEEL
58 Defl:N 51 12% 1980-01-01 00:00 9b5c4700 idna/__init__.py
@@ -284,7 +284,7 @@
4 Defl:N 6 -50% 1980-01-01 00:00 c2971fc7 idna-2.6.dist-info/INSTALLER
8866 Defl:N 3640 59% 1980-01-01 00:00 e49d9981 idna-2.6.dist-info/METADATA
1097 Defl:N 516 53% 1980-01-01 00:00 af1dfd18 idna-2.6.dist-info/metadata.json
- 1484 Defl:N 790 47% 1980-01-01 00:00 a82621a0 idna-2.6.dist-info/RECORD
+ 1484 Defl:N 791 47% 1980-01-01 00:00 a025dfb1 idna-2.6.dist-info/RECORD
5 Defl:N 7 -40% 1980-01-01 00:00 383ed7a7 idna-2.6.dist-info/top_level.txt
110 Defl:N 95 14% 1980-01-01 00:00 1382dbab idna-2.6.dist-info/WHEEL
101373 Defl:N 33527 67% 1980-01-01 00:00 6291aaa7 pkg_resources/__init__.py
@@ -307,7 +307,7 @@
4 Defl:N 6 -50% 1980-01-01 00:00 c2971fc7 pkg_resources-0.0.0.dist-info/INSTALLER
177 Defl:N 118 33% 1980-01-01 00:00 ea4f14c3 pkg_resources-0.0.0.dist-info/METADATA
221 Defl:N 168 24% 1980-01-01 00:00 35f04cfd pkg_resources-0.0.0.dist-info/metadata.json
- 2812 Defl:N 1100 61% 1980-01-01 00:00 33baba2b pkg_resources-0.0.0.dist-info/RECORD
+ 2812 Defl:N 1102 61% 1980-01-01 00:00 c845a63d pkg_resources-0.0.0.dist-info/RECORD
110 Defl:N 95 14% 1980-01-01 00:00 1382dbab pkg_resources-0.0.0.dist-info/WHEEL
34435 Defl:N 9280 73% 1980-01-01 00:00 a5b6ac27 pytz/__init__.py
1333 Defl:N 596 55% 1980-01-01 00:00 0047d14f pytz/exceptions.py
@@ -918,7 +918,7 @@
4 Defl:N 6 -50% 1980-01-01 00:00 c2971fc7 pytz-2017.2.dist-info/INSTALLER
20126 Defl:N 7645 62% 1980-01-01 00:00 f0b616fd pytz-2017.2.dist-info/METADATA
906 Defl:N 513 43% 1980-01-01 00:00 e0a03182 pytz-2017.2.dist-info/metadata.json
- 52688 Defl:N 21643 59% 1980-01-01 00:00 f69fed32 pytz-2017.2.dist-info/RECORD
+ 52688 Defl:N 21644 59% 1980-01-01 00:00 3be77323 pytz-2017.2.dist-info/RECORD
5 Defl:N 7 -40% 1980-01-01 00:00 4f3a402a pytz-2017.2.dist-info/top_level.txt
110 Defl:N 95 14% 1980-01-01 00:00 1382dbab pytz-2017.2.dist-info/WHEEL
1 Defl:N 3 -200% 1980-01-01 00:00 32d70693 pytz-2017.2.dist-info/zip-safe
@@ -944,7 +944,7 @@
4 Defl:N 6 -50% 1980-01-01 00:00 c2971fc7 requests-2.18.4.dist-info/INSTALLER
50496 Defl:N 20134 60% 1980-01-01 00:00 247dbf69 requests-2.18.4.dist-info/METADATA
1638 Defl:N 727 56% 1980-01-01 00:00 38843345 requests-2.18.4.dist-info/metadata.json
- 2849 Defl:N 1312 54% 1980-01-01 00:00 19a41c86 requests-2.18.4.dist-info/RECORD
+ 2849 Defl:N 1313 54% 1980-01-01 00:00 218d4c1f requests-2.18.4.dist-info/RECORD
9 Defl:N 11 -22% 1980-01-01 00:00 2ec0e227 requests-2.18.4.dist-info/top_level.txt
110 Defl:N 95 14% 1980-01-01 00:00 1382dbab requests-2.18.4.dist-info/WHEEL
2853 Defl:N 1195 58% 1980-01-01 00:00 87f53e92 urllib3/__init__.py
@@ -987,7 +987,7 @@
4 Defl:N 6 -50% 1980-01-01 00:00 c2971fc7 urllib3-1.22.dist-info/INSTALLER
32027 Defl:N 13222 59% 1980-01-01 00:00 d06cf4de urllib3-1.22.dist-info/METADATA
1653 Defl:N 718 57% 1980-01-01 00:00 4cb24210 urllib3-1.22.dist-info/metadata.json
- 5656 Defl:N 2306 59% 1980-01-01 00:00 8d6f40f4 urllib3-1.22.dist-info/RECORD
+ 5656 Defl:N 2292 60% 1980-01-01 00:00 702cfcc1 urllib3-1.22.dist-info/RECORD
8 Defl:N 10 -25% 1980-01-01 00:00 c052ff1b urllib3-1.22.dist-info/top_level.txt
110 Defl:N 95 14% 1980-01-01 00:00 1382dbab urllib3-1.22.dist-info/WHEEL
6842 Defl:N 2579 62% 1980-01-01 00:00 5b85439b werkzeug/__init__.py
@@ -1047,4 +1047,4 @@
10 Defl:N 12 -20% 1980-01-01 00:00 06727e3b xmltodict-0.11.0.dist-info/top_level.txt
110 Defl:N 95 14% 1980-01-01 00:00 1382dbab xmltodict-0.11.0.dist-info/WHEEL
-------- ------- --- -------
-17651153 7855116 56% 1045 files
+17651153 7855084 56% 1045 files
And I can confirm that stripping out the dist-info
makes the sha256 identical:
$ sha256sum a.zip b.zip
2d8719ad8b94d84f6535209a494a6608af333e67728163dd53ba97f1fcfa07b3 a.zip
61a795ad7c3659e3992f6c71f00c48eb9ae50849809b7a2c181ec6c880852664 b.zip
$ zip -d a.zip '*.dist-info/*'
...
$ zip -d b.zip '*.dist-info/*'
...
$ sha256sum a.zip b.zip
ad846ef5c8fd7d480e273d33f4ef1e2e5d09651658b54e9c80b32b89e3486702 a.zip
ad846ef5c8fd7d480e273d33f4ef1e2e5d09651658b54e9c80b32b89e3486702 b.zip
I'm getting the following error when I call "/" for the simple test app below
{"message":"Missing Authentication Token"}
api.py
from flask import Flask
app = Flask(__name__)
@app.route("/")
def index():
return "INDEX"
@app.route("/cats")
def cats():
return "Cats"
@app.route("/dogs/<id>")
def dog(id):
return "Dog = " + str(id)
serverless.yml
service: example
provider:
name: aws
runtime: python2.7
plugins:
- serverless-wsgi
functions:
api:
handler: wsgi.handler
events:
- http: ANY {proxy+}
custom:
wsgi:
app: api.app
Calls to /cats and /dogs/x work fine. Oddly, calling / from the API Gateway console returns successfully, but not when calling from curl or a web browser.
I am trying to setup a flask app using serverless-wsgi plugin on aws lambda. I am using api gateway proxy integration and have also enabled this content type in Binary Support. All my POST request's are getting through except for the ones with content-type multipart/form-data.
Does your plugin support Binary data via API gateway? If it does how would I get the contents of my file using flask request module:
Normally I can use below to save a file locally on my server:
data = request.files['customer_speech_file']
data.save('/tmp/' + str(data.filename))
Hi I have a flask app that returns content type application/zip
but when i run it using serverless-wsgi
it seems like the content type is changed to text/plain
. any idea what i might be doing wrong? Thanks
Hi,
I am trying to deploy a django graphql api to aws. I have exactly followed this example. In addition I have added the aws endpoint given after deploying to aws in allowed_hosts of the settings.py. Anyway, I always get "{"message": "Internal server error"}" independent of which view I am trying to access. According to serverless-wsgi the app under custom should be set to "cookbook.wsgi.application". But do I need to adjust the prefix of wsgi.handler under functions? Here it is "api", whereas in the official serverless blog post "Build a Python REST API with Serverless, Lambda, and DynamoDB" it is "app". Both are flask, to what should it be adjusted in this django case?
Currently my serverless.yml looks like this:
service: cookbook
plugins:
- serverless-python-requirements
- serverless-wsgi
custom:
wsgi:
app: cookbook.wsgi.application
packRequirements: false
pythonRequirements:
dockerizePip: non-linux
provider:
name: aws
runtime: python3.6
stage: dev
region: eu-central-1
functions:
app:
handler: wsgi.handler
events:
- http: ANY /
- http: 'ANY {proxy+}'
Cheers
I'm trying to use this on a Django project which doesn't use werkzeug
. In order for it to work, werkzeug
is a required dependency. Without installing werkzeug I received:
Unable to import module 'wsgi': No module named werkzeug.datastructures
I've directory structure as following
โโโ app
โย ย โโโ common
โย ย โย ย โโโ _
โย ย โย ย
| โโโ __init__.py
โโโ serverless.yml
(Some part omitted)
__init__.py
contains a global variable app
which flask application.
In serverless.yml
, I've configured the app as following app
functions:
api:
handler: wsgi.handler
events:
- http: ANY {proxy+}
custom:
wsgi:
app: app.__init__.app
There is one Flask API written as
@app.route('/api/v1/status', methods=['GET'])
def application_status():
"""
Application status
:return: success
"""
After deploying, I get the following endpoint
endpoints:
ANY - https://9vxxx7axx0.execute-api.us-east-1.amazonaws.com/dev/{proxy+}
When I try to send GET request on this endpoint
http://9xxxx7axx0.execute-api.us-east-1.amazonaws.com/dev/api/v1/status
I get the following output:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
What am I missing?
Hi.
I see the following error in my Lambda function Cloudwatch Logs when using this serverless plugin.
START RequestId: f5509b02-3c75-11e7-8d94-d97b07264760 Version: $LATEST
Syntax error in module 'wsgi': Missing parentheses in call to 'print' (wsgi.py, line 118)
END RequestId: f5509b02-3c75-11e7-8d94-d97b07264760
It seems that the print statement without parenthesis is not compatible with Python 3.6.
Any chance you'll be porting the plugin to a 3.6 compatible version soon?
Has file uploads / binary data been tested with this? Last I've tried, I was unable to get binary data through api gateway.
I can only get this to work on Centos 7. On any other OS it never creates the .requirements folder.
#As Root
yum install nodejs
yum install npm
npm install -g npm@lts
npm install -g serverless
#Local
mkdir web
cd web/
virtualenv web
source web/bin/activate
pip install flask
npm install --save serverless-wsgi
pip install werkzeug
pip install virtualenv
serverless create --template aws-python
vi serverless.yml
vi requirements.txt (Flask==0.11.1)
vi api.py (same code as readme)
serverless deploy
Everything is working on CentOS 7.x. If I run the exact same sample code on an Amazon Linux instance the .requirements folder is missing from the zip file. I ran the python requirements.py requirements.txt .requirements
command and it just creates and empty .requirements folder.
One thing that had to be done was to run yum remove python27-virtualenv
. This fixed issues with the serverless framework. I believe the issue is related to some pip package issue. I have listed the pre-installed pip packeages on the system below.
aws-cfn-bootstrap (1.4)
awscli (1.11.83)
Babel (0.9.4)
backports.ssl-match-hostname (3.4.0.2)
boto (2.42.0)
botocore (1.5.46)
chardet (2.0.1)
cloud-init (0.7.6)
colorama (0.2.5)
configobj (4.7.2)
docutils (0.11)
ecdsa (0.11)
futures (3.0.3)
gyp (0.1)
iniparse (0.3.1)
jmespath (0.9.2)
jsonpatch (1.2)
jsonpointer (1.0)
kitchen (1.1.1)
lockfile (0.8)
paramiko (1.15.1)
PIL (1.1.6)
pip (9.0.1)
ply (3.4)
pyasn1 (0.1.7)
pycrypto (2.6.1)
pycurl (7.19.0)
pygpgme (0.3)
pyliblzma (0.5.3)
pystache (0.5.3)
python-daemon (1.5.2)
python-dateutil (2.1)
pyxattr (0.5.0)
PyYAML (3.10)
requests (1.2.3)
rsa (3.4.1)
setuptools (12.2)
simplejson (3.6.5)
six (1.8.0)
urlgrabber (3.10)
urllib3 (1.8.2)
virtualenv (15.1.0)
yum-metadata-parser (1.1.4)
Is there a way to run CLI commands on the deployed application? A primary example of where this would be needed would be invoking a "manage" command for django, like collectstatic.
Something like serverless manage "collectstatic --no-input"
would be ideal.
As it stands, running commands like this is not possible using serverless-wsgi, or at least isn't clearly defined how to do so.
When I use this include/exclude configuration:
package:
exclude:
- '**' # everything
include:
- '*.py'
....requirements
and .wsgi_app
aren't packaged. I have to use this:
package:
exclude:
- '**' # everything
include:
- '*.py'
- '.requirements/**' # Serverless WSGI
- '.wsgi_app' # Serverless WSGI
It seems useful to include .requirements/**
and .wsgi_app
, even when that's not specified. In the same vein, it seems useful to exclude node_modules/serverless-wsgi/**
. (Similarly, Serverless itself excludes .serverless\**
, even when that's not specified.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.