Newline Pending Merge

    serverless-python-individually-deep

    0.1.9 • Public • Published

    serverless-python-individually

    serverless

    What's it?

    It's a simple plugin for serverless 1.3 that makes it easier to package multiple lambda functions written in python.

    Why do I need it?

    Say you have multiple lambda functions and each of them has fairly different package requirements. It's not economical to pack all dependencies in one big fat zip. Instead, this plugin can help to pack lambda functions with their own dependencies if you create requirements.txt for every function:

    project
    ├── hello
    │   ├── handler.py
    │   └── requirements.txt
    ├── world
    │   ├── handler.py
    │   └── requirements.txt
    └── serverless.yml
    

    That way, this plugin can help to pack lambda functions with their own dependencies.

    Moreover, if you are on a Mac, thanks to @docker-lambda, it can pull packages for Linux x86_64 too. More on that please read How to install platform-dependent packages.

    How?

    Be sure that virtualenv is installed. Otherwise,

    pip install virtualenv

    Then,

    npm install serverless-python-individually

    Your original serverless.yml may look like:

    functions:
      helloFunc:
        handler: hello/handler.hello
      worldFunc:
        handler: world/handler
    

    The plugin works by replacing the real handlers(e.g. hello/handler.hello) with a wrapper generated on the fly(e.g. hello/wrap.handler). The real handlers are instead set in custom.pyIndividually section.

    A modification to serverless.yml is needed:

    package:
      individually: True
      exclude:
        # Exclude everything first.
        - '**/*'
    functions:
      helloFunc:
        handler: hello/wrap.handler
        package:
          include:
            - hello/**
      worldFunc:
        handler: world/wrap.handler
        package:
          include:
            - world/**
    custom:
      pyIndividually:
        wrap:helloFunc: hello/handler.hello     # mapping to the real handler
        wrap:worldFunnc: world/handler.world    # mapping to the real handler
    
    plugins:
      - serverless-python-individually
    

    After sls deploy, you end up having many .zip in .serverless/. They are the actual artifacts that got uploaded to AWS Lambda by serverless. You can examine their content like:

    > tar tvzf .serverless/aws-python-dev-helloFunc.zip
    
    hello/handler.py
    hello/requirements.txt
    hello/wrap.py
    hello/lib/pkg_resources/...
    hello/lib/requests-2.12.3.dist-info/...
    hello/lib/requests/...
    hello/lib/...
    
    

    Notice that wrap.py and lib/ are created for you. All dependencies should have been pulled and installed in lib/. This plugin also works for sls deploy function -f.

    How to install platform-dependent packages

    If you are on a Mac, there're platform-dependent dependencies like subprocess32, bcrypt, etc., cannot simply be pip installed. One way to get around is to launch a aws-lambda architecture identical EC2 or a VM to do the job. That's inconvenient to say the least. Thanks to @docker-lambda, we can launch a container for the same purpose at our disposal. All you need to do is:

    • Make sure docker is installed and properly set up. I.e. when running docker version you should see information about client and server.
    • docker pull lambci/lambda:build-python2.7 to pull the image in advance.
    • Turn on dockerizedPip in serverless.yml:
      custom:
          pyIndividually:
              # ...
      
              # Launches a container for installing packages.
              # The default is False.
              dockerizedPip: True
      

    Advanced configuration

    There are a couple of configurations that can be handy for you.

    severless.yml

    • wrap.py and lib/ are created during packaging in the same directory where the real handler is. If you are not happy about the naming, you can change wrapName and libSubDir.
    • wrap.py and lib/ by default will be deleted after packaging. They can be preserved by setting cleanup to False.
    custom:
      pyIndividually:
        # A ${wrapName}.py will be generated for every function.
        # The default filename is 'wrap.py', but you can change it to avoid name clashes.
        wrapName: wrapFoo
    
        # pip install packages to ${libSubDir} along with ${wrapNam}.py
        # The default dir is 'lib'.
        # libSubDir: lib
    
        # Cleanup ${libSubDir} and ${wrapName}.py created by the plugin.
        # The default is True.
        # cleanup: True
    
        # Mapping to the real handler of every function. In the format:
        # ${wrapName}:function_name: real_handler
        # If there's no mapping for a function, then that function will not be touced by this plugin.
        wrapFoo:helloFunc: hello/handler.hello
        wrapFoo:worldFunnc: world/handler.world
    
        # See [How to install platform-dependent package]
        # The default is False.
        # dockerizedPip: False
    
    

    Command line options

    You can also overwrite some configurations through extra options when sls deploy.

    • --pi-cleanup/--pi-no-cleanup overwrite cleanup in serverless.yml.

    • --pi-dockerizedPip/--pi-no-dockerizedPip overwrite dockerizedPip in serverless.yml.

    • --pi-disable skips this plugin.

    • Handy but USE WITH CAUTION: If --pi-no-cleanup was specified previously and you don't want to pull dependencies again, then you can disable this plugin temporarily with --pi-disable. sls would pack what's left over in the directory:

    $> sls deploy --pi-no-cleanup
    Now wrap.py and lib/* are not cleaned. You can do some work. Make sure requirements.txt not being changed anyhow.
    $> sls deploy --pi-disable
    The plugin is disabled for this time. sls should then directly pack wrap.py and lib/* left last time.
    

    Demo

    A demo is there for you to get started.

    Credit

    This plugin is heavily influenced by serverless-wsgi from @logandk. In fact, the requirement installer is directly borrowed from his repo. If your lambda is a wsgi app, then must check out his work.

    Also thanks to @docker-lambda to provide aws lambda runtime equivalent docker image.

    Note

    As of this writing, I just start using serverless 1.3. This plugin may or may not work with other 1.x versions but I haven't tried.

    Install

    npm i serverless-python-individually-deep

    DownloadsWeekly Downloads

    18

    Version

    0.1.9

    License

    MIT

    Last publish

    Collaborators

    • daniel.bardsley