Precompiled packages for AWS lambda
- https://aws.amazon.com/lambda/ and create/sign in into account
- Lambda > Functions - Create lambda function
- Black function
- Configure triggers - Next
- Configure function
- Runtime - Python 2.7
- Lambda function handler and role
- Handler - service.handler
- Role - Create new role from template(s)
- Role name - test
- Policy templates - Simple Microservice Permissions
- Advanced settings
- Memory (MB) 128
- Timeout 1 min 0 sec
- Code entry type - Upload a .ZIP file - choose Pack.zip from rep
- Test -> Save and test
- Modify service.py file from sources folder
- Choose all files in sources folder when compressing, don't put it in one folder
- Upload zip file on function page
Selenium on PhantomJS. In fact - a ready-made tool for web scraping. For example, the demo now opens a random page in Wikipedia and sends its header. (PhantomJS at the same time disguises itself as a normal browser, knows how to log in, click and fill out forms) Also added requests, so you can do API requests for different resources to discard / take away the information.
Useful for web testing and scraping.
Current demo opens random page from wiki (https://en.wikipedia.org/wiki/Special:Random) and prints title.
https://selenium-python.readthedocs.io/
What does the lambda have to do with it? In a nutshell on AWS in one region you can simultaneously run 200 lambdas (more if you write to support). Lambda works in 11 regions. So you can run in parallel more than 2000 lambdas, each of which will conduct load testing of your service. Five minutes of such testing will cost just one dollar.
Demo in this package tries to send requests to github.com for 5 seconds with 1 connection and also conduct pyresttest dummy test.
- WRK (https://github.com/wg/wrk) - the main tool for load testing. It works with multiple threads, you can specify the number of connections and length of the load. For more fine-tuning, you can use LuaJIT scripts (https://www.lua.org/).
- Pyrestest (https://github.com/svanoort/pyresttest) is a handy tool for testing the full pipeline of the API. For example, the user registers, then uses the api key to create tasks / make notes / downloads files, then reads them, then deletes them.
https://github.com/svanoort/pyresttest
Package for parsing static HTML pages. Difference here is that it works faster and consumes less memory than PhantomJS but is limited in terms websites it can parse and other features.
serverless install -u https://github.com/ryfeus/lambda-packs/tree/master/Lxml_requests/source -n lxml-requests
cd lxml-requests
serverless deploy
serverless invoke --function main --log
wget https://github.com/ryfeus/lambda-packs/blob/master/Lxml_requests/buildPack.sh
docker pull amazonlinux:latest
docker run -v $(pwd):/outputs --name lambdapackgen -d amazonlinux:latest tail -f /dev/null
docker exec -i -t lambdapackgen /bin/bash /outputs/buildPack.sh
Lxml 3.7.1
Open source library for Machine Intelligence. Basically revolutionized AI and made it more accessible. Using tensorflow on lambda is not as bad as it may sound - for some simple models it is the simplest and the cheapest way to deploy.
As hello world code I used recognition of images trained on imagenet (https://www.tensorflow.org/tutorials/image_recognition). Given the price tag lambda one run (recognition of one picture) will cost $0.00005. Therefore for a dollar you can recognize 20,000 images. It is much cheaper than almost any alternatives, though completely scalable (200 functions can be run in parallel), and can be easily integrated into cloud infrastructure. Current demo downloads image from link 'imagelink' from event source ( if empty - then downloads https://s3.amazonaws.com/ryfeuslambda/tensorflow/imagenet/cropped_panda.jpg)
Tensorflow 1.4.0
https://www.tensorflow.org/tutorials/image_recognition
serverless install -u https://github.com/ryfeus/lambda-packs/tree/master/tensorflow/source -n tensorflow
cd tensorflow
serverless deploy
serverless invoke --function main --log
wget https://github.com/ryfeus/lambda-packs/blob/master/tensorflow/buildPack.sh
wget https://github.com/ryfeus/lambda-packs/blob/master/tensorflow/index.py
docker pull amazonlinux:latest
docker run -v $(pwd):/outputs --name lambdapackgen -d amazonlinux:latest tail -f /dev/null
docker exec -i -t lambdapackgen /bin/bash /outputs/buildPack.sh
Package for fans of machine learning, building models and the like. I doubt that there is a more convenient way to deploy model to the real world.
- Scikit-learn 0.17.1
- Scipy 0.17.0
Package of image processing tools, and not only to style image, but also a large set of computer vision algorithms.
Scikit-image 0.12.3
Another package of image processing tools, and not only to style image, but also a large set of Computer vision algorithms.
- OpenCV 3.1.0
- PIL 4.0.0
https://pillow.readthedocs.io/
http://docs.opencv.org/3.0-beta/doc/py_tutorials/py_tutorials.html
Package for fans of statistics, data scientists and data engineers. RAM at lambda is 1.5 gigabytes, and the maximum operating time - 5 minutes. I am sure that will be enough for most tasks.
Pandas 0.19.0
PDF generator + Microsoft office file generator (docx, xlsx, pptx) + image generator (jpg, png) + book generator (epub)
"Hello world" code in package creates example of every document. Basically these libs are low memory (less than 128MB) and high speed (less than 0.5 seconds) so it's something like ~1m documents generated per 1$ in terms of AWS Lambda pricing.
- docx (python-docx - https://pypi.python.org/pypi/python-docx)
- xlsx (XlsxWriter - https://pypi.python.org/pypi/XlsxWriter)
- pptx (python-pptx - https://pypi.python.org/pypi/python-pptx)
- pdf (Reportlab - https://pypi.python.org/pypi/reportlab)
- epub (EbookLib - https://pypi.python.org/pypi/EbookLib)
- png/jpg/... (Pillow - https://pypi.python.org/pypi/Pillow)
AWS Lambda pack in Python for processing satellite imagery. Basically it enables to deploy python code in an easy and cheap way for processing satellite imagery or polygons. In “hello world” code of the pack I download red, green, blue Landsat 8 bands from AWS, make True Color image out of it and upload it to S3. It takes 35 seconds and 824MB of RAM for it so ~2500 scenes can be processed for 1$.
- Rasterio (https://github.com/mapbox/rasterio 0.36)
- OSGEO (https://trac.osgeo.org/gdal/wiki/GdalOgrInPython)
- Pyproj (https://github.com/jswhit/pyproj)
- Shapely (https://github.com/Toblerity/Shapely)
- PIL (https://pillow.readthedocs.io/)