Serverless Python Requirements
A Serverless v1.x plugin to automatically bundle dependencies from
requirements.txt and make them available in your
Requires Serverless >= v1.12
sls plugin install -n serverless-python-requirements
This will automatically add the plugin to your project's
package.json and the plugins section of its
serverless.yml. That's all that's needed for basic use! The plugin will now bundle your python
dependencies specified in your
Pipfile when you run
For a more in depth introduction on how to user this plugin, check out this post on the Serverless Blog
If you're on a mac, check out these notes about using python installed by brew.
Compiling non-pure-Python modules or fetching their manylinux wheels is
supported on non-linux OSs via the use of Docker and the
To enable docker usage, add the following to your
The dockerizePip option supports a special case in addition to booleans of
'non-linux' which makes
it dockerize only on non-linux environments.
To utilize your own Docker container instead of the default, add the following to your
custom:pythonRequirements:dockerImage: <image name>:tag
This must be the full image name and tag to use, including the runtime specific tag if applicable.
Alternatively, you can define your Docker image in your own Dockerfile and add the following to your
Dockerfile the path to the Dockerfile that must be in the current folder (or a subfolder).
Please note the
dockerImage and the
dockerFile are mutually exclusive.
To install requirements from private git repositories, add the following to your
custom:pythonRequirements:dockerizePip: truedockerSsh: true
dockerSsh option will mount your
$HOME/.ssh/known_hosts as a
volume in the docker container. If your SSH key is password protected, you can use
$SSH_AUTH_SOCK is also mounted & the env var set.
It is important that the host of your private repositories has already been added in your
$HOME/.ssh/known_hosts file, as the install process will fail otherwise due to host authenticity
Pipenv support ✨🍰✨
If you include a
Pipfile and have
pipenv installed instead of a
requirements.txt this will use
pipenv lock -r to generate them. It is fully compatible with all options such as
dockerizePip. If you don't want this plugin to generate it for you, set the following option:
Dealing with Lambda's size limitations
To help deal with potentially large dependencies (for example:
scikit-learn) there is support for compressing the libraries. This does
require a minor change to your code to decompress them. To enable this add the
following to your
and add this to your handler module before any code that imports your deps:
try:import unzip_requirementsexcept ImportError:pass
Works on non 'win32' environments: Docker, WSL are included
To remove the tests, information and caches from the installed packages, enable the
slim option. This will:
.so files, remove
Custom Removal Patterns
To specify additional directories to remove from the installed packages,
define a list of of patterns int he serverless config using the
option and glob syntax. Note, it matches against whole paths, so to match a file in any
directory, start your pattern with
custom:pythonRequirements:slim: trueslimPatterns:- "**/*.egg-info*"
This will remove all folders within the installed requirements that match
the names in
You can omit a package from deployment with the
noDeploy option. Note that
dependencies of omitted packages must explicitly be omitted too.
By default, this will not install the AWS SDKs that are already installed on
Lambda. This example makes it instead omit pytest:
Extra Config Options
You can enable two kinds of caching with this plugin which are currently both DISABLED by default. First, a download cache that will cache downloads that pip needs to compile the packages. And second, a what we call "static caching" which caches output of pip after compiling everything for your requirements file. Since generally requirements.txt files rarely change, you will often see large amounts of speed improvements when enabling the static cache feature. These caches will be shared between all your projects if no custom cacheLocation is specified (see below).
Please note: This has replaced the previously recommended usage of "--cache-dir" in the pipCmdExtraArgs
custom:pythonRequirements:useDownloadCache: trueuseStaticCache: true
Additionally, In future versions of this plugin, both caching features will probably be enabled by default
Other caching options...
There are two additional options related to caching. You can specify where in your system that this plugin caches with the
cacheLocation option. By default it will figure out automatically where based on your username and your OS to store the cache via the appdirectory module. Additionally, you can specify how many max static caches to store with
staticCacheMaxVersions, as a simple attempt to limit disk space usage for caching. This is DISABLED (set to 0) by default. Example:
custom:pythonRequirements:useStaticCache: trueuseDownloadCache: truecacheLocation: '/home/user/.my_cache_goes_here'staticCacheMaxVersions: 10
Extra pip arguments
You can specify extra arguments supported by pip to be passed to pip like this:
Customize requirements file name
pip workflows involve using requirements files not named
To support these, this plugin has the following option:
If you have different python functions, with different sets of requirements, you can avoid including all the unecessary dependencies of your functions by using the following structure:
├── serverless.yml ├── function1 │ ├── requirements.txt │ └── index.py └── function2 ├── requirements.txt └── index.py
With the content of your
package:individually: truefunctions:func1:handler: index.handlermodule: function1func2:handler: index.handlermodule: function2
The result is 2 zip archives, with only the requirements for function1 in the first one, and only the requirements for function2 in the second one.
Quick notes on the config file:
modulefield must be used to tell the plugin where to find the
requirements.txtfile for each function.
handlerfield must not be prefixed by the folder name (already known through
module) as the root of the zip artifact is already the path to your function.
Customize Python executable
Sometimes your Python executable isn't available on your
python3.6 (for example, windows or using pyenv).
To support this, this plugin has the following option:
Vendor library directory
For certain libraries, default packaging produces too large an installation,
even when zipping. In those cases it may be necessary to tailor make a version
of the module. In that case you can store them in a directory and use the
vendor option, and the plugin will copy them along with all the other
dependencies to install:
custom:pythonRequirements:vendor: ./vendored-librariesfunctions:hello:handler: hello.handlervendor: ./hello-vendor # The option is also available at the function level
requirements.zip(if using zip support) files are left
behind to speed things up on subsequent deploys. To clean them up, run
sls requirements clean. You can also create them (and
using zip support) manually with
sls requirements install.
Invalidate requirements caches on package
If you are using your own Python library, you have to cleanup
.requirements on any update. You can use the following option to cleanup
.requirements everytime you package.
custom: pythonRequirements: invalidateCaches: true
🍎🍺🐍 Mac Brew installed Python notes
Brew wilfully breaks the
--target option with no seeming intention to fix it
which causes issues since this uses that option. There are a few easy workarounds for this:
- Create a virtualenv and activate it while using serverless.
Also, brew seems to cause issues with pipenv, so make sure you install pipenv using pip.
For usage of
dockerizePip on Windows do Step 1 only if running serverless on windows, or do both Step 1 & 2 if running serverless inside WSL.
- Enabling shared volume in Windows Docker Taskbar settings
- Installing the Docker client on Windows Subsystem for Linux (Ubuntu)
Native Code Dependencies During Build
Some Python packages require extra OS dependencies to build successfully. To deal with this, replace the default image (
lambci/lambda:python3.6) with a
# AWS Lambda execution environment is based on Amazon Linux 1FROM amazonlinux:1# Install Python 3.6RUN yum -y install python36 python36-pip# Install your dependenciesRUN curl -s | python3RUN yum -y install python3-devel mysql-devel gcc# Set the same WORKDIR as default imageRUN mkdir /var/taskWORKDIR /var/task
Then update your
Native Code Dependencies During Runtime
Some Python packages require extra OS libraries (
*.so files) at runtime. You need to manually include these files in the root directory of your Serverless package. The simplest way to do this is to commit the files to your repository:
For instance, the
mysqlclient package requires
libmysqlclient.so.1020. If you use the Dockerfile from the previous section, you can extract this file from the builder Dockerfile:
- Extract the library:
docker run --rm -v "$(pwd):/var/task" sls-py-reqs-custom cp -v /usr/lib64/mysql57/libmysqlclient.so.1020 .
(If you get the error
Unable to find image 'sls-py-reqs-custom:latest' locally, run
sls package to build the image.)
2. Commit to your repo:
git add libmysqlclient.so.1020git commit -m "Add libmysqlclient.so.1020"
- Verify the library gets included in your package:
sls packagezipinfo .serverless/xxx.zip
(If you can't see the library, you might need to adjust your package include/exclude configuration in
- @dschep - Lead developer & maintainer
- @azurelogic - logging & documentation fixes
- @abetomo - style & linting
- @angstwad -
- @mather - the cache invalidation option
- @rmax - the extra pip args option
- @bsamuel-ui - Python 3 support
- @suxor42 - fixing permission issues with Docker on Linux
- @mbeltran213 - fixing docker linux -u option bug
- @Tethik - adding usePipenv option
- @miketheman - fixing bug with includes when using zip option
- @wattdave - fixing bug when using
- @heri16 - fixing Docker support in Windows
- @ryansb - package individually support
- @cgrimal - Private SSH Repo access in Docker,
dockerFileoption to build a custom docker image, real per-function requirements, and the
- @kichik - Imposed windows &
noDeploysupport, switched to adding files straight to zip instead of creating symlinks, and improved pip chache support when using docker.
- @dee-me-tree-or-love - the
- @alexjurkiewicz - docs about docker workflows
- @andrewfarley - Implemented download caching and static caching