cfchou / serverless-python-individually

A serverless framework plugin to install multiple lambda functions written in python.

serverless-python-individually

• What's it?
• What's new?
• Why do I need it?
• How?
• How to install platform-dependent packages?
• Advanced configuration
• Demo
• Credit
• Note

What's it?

It's a simple plugin for serverless 1.3+ that makes it easier to package multiple lambda functions written in python.

What's new?

• Since 0.1.6 python3.6 is supported.
• Since 0.1.5 lambda functions placed under subdirectories are supported.

Why do I need it?

Say you have multiple lambda functions and each of them has fairly different package requirements. It's not economical to pack all dependencies in one big fat zip. Instead, this plugin can help to pack lambda functions with their own dependencies if you create requirements.txt for every function:

project
├── hello
│   ├── handler.py
│   └── requirements.txt
├── world
│   ├── handler.py
│   └── requirements.txt
└── serverless.yml

That way, this plugin can help to pack lambda functions with their own dependencies.

Moreover, if you are on a Mac, thanks to @docker-lambda, it can pull packages for Linux x86_64 too. More on that please read How to install platform-dependent packages.

How?

Be sure that virtualenv is installed. Otherwise,

pip install virtualenv

Then,

npm install serverless-python-individually

Your original serverless.yml may look like:

functions:
  helloFunc:
    handler: hello/handler.hello
  worldFunc:
    handler: world/handler.world

The plugin works by replacing the real handlers(e.g. hello/handler.hello) with a wrapper generated on the fly(e.g. hello/wrap.handler). The real handlers are instead set in custom.pyIndividually section.

A modification to serverless.yml is needed:

package:
  individually: True
  exclude:
    # Exclude everything first.
    - '**/*'
functions:
  helloFunc:
    handler: hello/wrap.handler
    package:
      include:
        - hello/**
  worldFunc:
    handler: world/wrap.handler
    package:
      include:
        - world/**
custom:
  pyIndividually:
    wrap:helloFunc: hello/handler.hello     # mapping to the real handler
    wrap:worldFunnc: world/handler.world    # mapping to the real handler

plugins:
  - serverless-python-individually

After sls deploy, you end up having many .zip in .serverless/. They are the actual artifacts that got uploaded to AWS Lambda by serverless. You can examine their content like:

> tar tvzf .serverless/aws-python-dev-helloFunc.zip

hello/handler.py
hello/requirements.txt
hello/wrap.py
hello/lib/pkg_resources/...
hello/lib/requests-2.12.3.dist-info/...
hello/lib/requests/...
hello/lib/...

Notice that wrap.py and lib/ are created for you. All dependencies should have been pulled and installed in lib/. This plugin also works for sls deploy function -f.

How to install platform-dependent packages

If you are on a Mac, there're platform-dependent dependencies like subprocess32, bcrypt, etc., cannot simply be pip installed. One way to get around is to launch a aws-lambda architecture identical EC2 or a VM to do the job. That's inconvenient to say the least. Thanks to @docker-lambda, we can launch a container for the same purpose at our disposal. All you need to do is:

• Make sure docker is installed and properly set up. I.e. when running docker version you should see information about client and server.
• For python2.7, docker pull lambci/lambda:build-python2.7 to pull the image in advance.
• For python3.6, docker pull lambci/lambda:build-python3.6 to pull the image in advance.
• Turn on dockerizedPip in serverless.yml:

  custom:
      pyIndividually:
          # ...
  
          # Launches a container for installing packages.
          # The default is False.
          dockerizedPip: True
  

Advanced configuration

There are a couple of configurations that can be handy for you.

severless.yml

• wrap.py and lib/ are created during packaging in the same directory where the real handler is. If you are not happy about the naming, you can change wrapName and libSubDir.
• wrap.py and lib/ by default will be deleted after packaging. They can be preserved by setting cleanup to False.

custom:
  pyIndividually:
    # A ${wrapName}.py will be generated for every function.
    # The default filename is 'wrap.py', but you can change it to avoid name clashes.
    wrapName: wrapFoo

    # pip install packages to ${libSubDir} along with ${wrapNam}.py
    # The default dir is 'lib'.
    # libSubDir: lib

    # Cleanup ${libSubDir} and ${wrapName}.py created by the plugin.
    # The default is True.
    # cleanup: True

    # Mapping to the real handler of every function. In the format:
    # ${wrapName}:function_name: real_handler
    # If there's no mapping for a function, then that function will not be touced by this plugin.
    wrapFoo:helloFunc: hello/handler.hello
    wrapFoo:worldFunnc: world/handler.world

    # See [How to install platform-dependent package]
    # The default is False.
    # dockerizedPip: False

Command line options

You can also overwrite some configurations through extra options when sls deploy.

• --pi-cleanup/--pi-no-cleanup overwrite cleanup in serverless.yml.

• --pi-dockerizedPip/--pi-no-dockerizedPip overwrite dockerizedPip in serverless.yml.

• --pi-disable skips this plugin.

• Handy but USE WITH CAUTION: If --pi-no-cleanup was specified previously and you don't want to pull dependencies again, then you can disable this plugin temporarily with --pi-disable. sls would pack what's left over in the directory:

$> sls deploy --pi-no-cleanup
Now wrap.py and lib/* are not cleaned. You can do some work. Make sure requirements.txt not being changed anyhow.
$> sls deploy --pi-disable
The plugin is disabled for this time. sls should then directly pack wrap.py and lib/* left last time.

Demo

A demo is there for you to get started.

Credit

This plugin is heavily influenced by serverless-wsgi from @logandk. In fact, the requirement installer is directly borrowed from his repo. If your lambda is a wsgi app, then must check out his work.

Also thanks to @docker-lambda to provide aws lambda runtime equivalent docker image.

Note

As of this writing, I just start using serverless 1.3+. This plugin may or may not work with other 1.x versions but I haven't tried.

serverless-python-individually's Issues

Only requirements.txt is displayed wrong

Overview

The requirements.txt in the documentation is not correct.

Details

Ref: https://www.serverless.com/plugins/serverless-python-individually

Environment

• Windows 11 Pro
• Chrome
• 12:20, 1/14/2024 (JST)

Subjectivity

I think it is possible that only the documentation for this plugin is displayed incorrectly, but that the front end of the plugin documentation is actually incorrect.

Lambda's size limitations (zip the lib forlder when deploying)

Hello,
thanks for this awesome work,
is it possible to add an option to zip/unzip lib folder when deploying to avoid size limitation of lambda like in here : https://github.com/UnitedIncome/serverless-python-requirements#dealing-with-lambdas-size-limitations

Problem with new version of pip

We started to have problems using the plugin with 'dockerizedPip: True' since yesterday (19/03/2018).

After some search we've found the cause of this error:

serverless-python-individually/index.js

Line 319 in 7c396f5

if (ret.error || ret.stderr.length != 0) {

This check is causing problem when a new version of pip is released and the docker image is not updated with that version. Because pip prints the warning message of new version available in the stderr.
Is there a reason to check for the size of stderr? The 'ret.error' isn't enough?

[pyIndividually] Unable to load virtualenv, please install

How to resolve this ?

unable to import module 'hello/wrap'

Followed the example, but I get

{
  "errorMessage": "Unable to import module 'hello/wrap'"
}

python2.7

python 2.7 deprecation warning causes error

Since today my deployments are failing with the error Unhandled error in pip, not deploying to AWS..
Also I see a new warning in the logs:
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

This issue seems to be similar to #13

Virtualenv 20.x is not supported

I had to downgrade to virtualenv 16.7.10 to get this to work, otherwise I get the following error when I attempt a sls package:

Serverless: [pyIndividually] Installing packagings: python testproject/lib/_requirements.py testproject/requirements.txt testproject/lib
Serverless: [pyIndividually] Traceback (most recent call last):
  File "testproject/lib/_requirements.py", line 57, in <module>
    virtualenv.main()

serverless.yml

service: testproject
frameworkVersion: "2.11.1"

package:
    individually: True
    exclude:
        - "**/*"
plugins:
    - serverless-python-individually

provider:
    name: aws
    runtime: python3.6
    region: us-west-2

functions:
    testproject:
        handler: testproject/wrap.handler
        package:
            include:
                - testproject/**
custom:
    pyIndividually:
        wrap:testproject: testproject/runner.handle_message

Issues with running in Travis CI

Hi @cfchou,

I faced this issue when I was trying to configure Travis CI to deploy my project. Serverless Framework failed with this error

  File "gateway/lib/_requirements.py", line 56, in <module>
    sys_stdout.write(b'creating virtualenv {}\n'.format(VENV_DIR))
AttributeError: 'bytes' object has no attribute 'format'

This issue is reproducible locally as well.

virtualenv -p python3.6 venv
pip install -t lib/ -r requirements.txt
. ./venv/bin/activate
serverless deploy

Created PR #2.

Does not work in subdirectories

When I try to run a deployed Lambda function I get the following error:
Syntax error in module 'src/stepfunction/cropping/wrap': invalid syntax (wrap.py, line 11)

The created wrap.py function looks like this:

# vim:fileencoding=utf-8
# wrap.py
# This file is generated on the fly by serverless-python-individually plugin.
import os
import sys

root = os.path.abspath(os.path.join(os.path.dirname(__file__)))
sys.path[0:0] = [root, os.path.join(root, "lib")]

from src/stepfunction/cropping/cropDinA4 import lambda_handler as real_handler # THIS LINE!!!

def handler(event, context):
  return real_handler(event, context)

I guess this is due to the fact that the function is in a subdirectory.
Is there a way that can be supported?

how to compile OpenCV (cv2)

I'm sorry to bother you with this question. I'm not a python developer I just need to integrate some python code as lambda functions into my serverless configuration.

I was provided with precompiled OpenCV that targets Lambda but I thought it would be better if that can be integrated into the serverless deployment step.

For numpy I was able to create the requirements.txt just fine but OpenCV doesn't seems to be installable via pip. I know that OpenCV has to be compiled and I read that this serverless plugin is able to do that via running a Docker container. I configured the plugin to run Docker but now I'm stuck.
How can I build OpenCV?

Cleanup fails when using dockerizedPip

Problem
When using the dockerizedPip packaging, the plugin starts docker to install pip dependencies. Since docker runs as root, the owner of the installed packages is also root. On subsequent runs, the plugin fails as it cannot delete files owned by root, cleanup fails and so on.

Solution
Change the owner of the installed packages to the owner of the lib folder after the installation.

Hi, @cfchou! Faced this issue today. Fixed here #9