· python serverless

Serverless: Python - virtualenv - { "errorMessage": "Unable to import module 'handler'" }

I’ve been using the Serverless library to deploy and run some Python functions on AWS lambda recently and was initially confused about how to handle my dependencies.

I tend to create a new virtualenv for each of my project so let’s get that setup first:

Prerequisites

$ npm install serverless
$ virtualenv -p python3 a
$ . a/bin/activate

Now let’s create our Serverless project. I’m going to install the requests library so that I can use it in my function.

My Serverless project

serverless.yaml

service: python-starter-template

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: python3.6
  timeout: 180

functions:
  starter-function:
      name: Starter
      handler: handler.starter

handler.py

import requests

def starter(event, context):
    print("event:", event, "context:", context)
    r = requests.get("http://www.google.com")
    print(r.status_code)
$ pip install requests

Ok, we’re now ready to try out the function. A nice feature of Serverless is that it lets us try out functions locally before we deploy them onto one of the Cloud providers:

$ ./node_modules/serverless/bin/serverless invoke local --function starter-function
event: {} context: <__main__.FakeLambdaContext object at 0x10bea9a20>
200
null

So far so good. Next we’ll deploy our function to AWS. I’m assuming you’ve already got your credentials setup but if not you can follow the tutorial on the Serverless page.

$ ./node_modules/serverless/bin/serverless deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (26.48 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.........
Serverless: Stack update finished...
Service Information
service: python-starter-template
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  None
functions:
  starter-function: python-starter-template-dev-starter-function

Now let’s invoke our function:

$ ./node_modules/serverless/bin/serverless invoke --function starter-function
{
    "errorMessage": "Unable to import module 'handler'"
}

  Error --------------------------------------------------

  Invoked function failed

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           6.7.0
     Serverless Version:     1.19.0

Hmmm, that’s odd - I wonder why it can’t import our handler module? We can call the logs function to check. The logs are usually a few seconds behind so we’ll have to be a bit patient if we don’t see them immediately.

$ ./node_modules/serverless/bin/serverless logs  --function starter-function
START RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552 Version: $LATEST
Unable to import module 'handler': No module named 'requests'

END RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552
REPORT RequestId: 735efa84-7ad0-11e7-a4ef-d5baf0b46552	Duration: 0.42 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 22 MB

That explains it - the requests module wasn’t imported.

If we look in .serverless/python-starter-template.zip</p> we can see that the requests module is hidden inside the a directory and the instance of Python that runs on Lambda doesn’t know where to find it. </p>

I’m sure there are other ways of solving this but the easiest one I found is a Serverless plugin called serverless-python-requirements.

So how does this plugin work?

A Serverless v1.x plugin to automatically bundle dependencies from requirements.txt and make them available in your PYTHONPATH.

Doesn’t sound too tricky - we can use pip freeze to get our list of requirements and write them into a file. Let’s rework serverless.yaml to make use of the plugin:

My Serverless project using serverless-python-requirements ~bash $ npm install --save serverless-python-requirements ~ ~bash $ pip freeze > requirements.txt $ cat requirements.txt certifi==2017.7.27.1 chardet==3.0.4 idna==2.5 requests==2.18.3 urllib3==1.22 ~

serverless.yaml ~yaml service: python-starter-template frameworkVersion: ">=1.2.0 <2.0.0" provider: name: aws runtime: python3.6 timeout: 180 plugins: - serverless-python-requirements functions: starter-function: name: Starter handler: handler.starter package: exclude: - a/** # virtualenv ~

We have two changes from before:

  • We added the serverless-python-requirements plugin

  • We excluded the a directory since we don’t need it

Let’s deploy again and run the function: ~bash $ ./node_modules/serverless/bin/serverless deploy Serverless: Parsing Python requirements.txt Serverless: Installing required Python packages for runtime python3.6... Serverless: Linking required Python packages... Serverless: Packaging service... Serverless: Excluding development dependencies... Serverless: Unlinking required Python packages... Serverless: Uploading CloudFormation file to S3... Serverless: Uploading artifacts... Serverless: Uploading service .zip file to S3 (14.39 MB)... Serverless: Validating template... Serverless: Updating Stack... Serverless: Checking Stack update progress... ......... Serverless: Stack update finished... Service Information service: python-starter-template stage: dev region: us-east-1 api keys: None endpoints: None functions: starter-function: python-starter-template-dev-starter-function ~ ~bash $ ./node_modules/serverless/bin/serverless invoke --function starter-function null ~

Looks good. Let’s check the logs: ~bash $ ./node_modules/serverless/bin/serverless logs --function starter-function START RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24 Version: $LATEST event: {} context: <main.LambdaContext object at 0x7f568b105f28> 200 END RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24 REPORT RequestId: 61e8eda7-7ad4-11e7-8914-03b8a7793a24 Duration: 55.55 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 29 M ~

All good here as well so we’re done!

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket