Introduction
We will go over How To Create An AWS Python Lambda Layer.
Did you know AWS Python Lambda Layers allow you to include re-useable code in all your Lambdas?
I will break this review in the following sections:
- What is a python lambda layer and why you should use it
- How to create a lambda layer
- How to add Python requirements in your lambda layer
- How to import your Python Lambda Layer in your Lambda code
I have been personally using this setup across all my lambda deployments and it works like a charm.
We will go point by point on getting you up and running in less than 5mins of work.
This is a complete guide on Python AWS Lambda layers so you will find answers to all your common questions and problems here along with a full working example of a Lambda Layer Deployment.
All code associated with this can be found in the GitHub page here.
What Is A Python Lambda Layer
A Python Lambda Layer is basically a shared resource that you can attach to any of the Lambdas you have developed. Basically you can think of it as a library module that has all your boiler plate code, initialization functions along with helpers.
It can be versioned and tracked and attached accordingly. So you can keep multiple versions of the same layer and decide which one you want to associate with your Lambdas.
When You Should Use a Python Lambda Layer
You should use a Python Lambda Layer if you are intending on sharing code across your lambdas. If you are doing a single Lambda deployment then it’s not necessary but if you are going to be using more than one lambda and those may be having code components that are shared it’s recommended to be used.
From the above you can understand that there’s a lot of reasons why you should use it but there’s also a reason why you should not use it:
- Overcomplicate simple code without a reason
- You do not have any shared code
- You are only using one lambda
- There is no boiler plate code in your lambda
Now that we covered when you should use layers lets cover of how they could be useful for your Lambda.
Why Are AWS Python Lambda Layers Useful
Here are some reasons of why a Python Lambda Layer could be useful for your code:
- Keep codes clean
- You can separate your boiler plate code from your main code
- You can segregate and layer your code as needed
- You only have to debug and fix only once the common code
- You can add all the Python package requirements in the Layer
How To Create A Python Lambda Layer In AWSFirst we are going to cover how to create a Python Lambda Layer using the AWS web console. In order to do this first navigate to the Lambda tab either using the search bar at the top or following a quick shortcut.
Once you are in the Lambda section of AWS click on the Layer from the left hand side menu.
This should greet you with the screen as seen below:
Once you see the Layer screen go ahead and click on the Create Layer lambda button at the top right so we can get started on creating our Lambda.
There’s a couple of important things you need to pay attention too here:
- The Python Runtime version, make sure whatever you select here is basically compatible with the code you want to execute.
- The ZIP file contents of the actual lambda layer code. For the name of the ZIP file you can select whatever you want but the contents of it should be based on what we are going to talk about below.
For the ZIP file we will be using a very simple lambda layer for our example.
How To Create Lambda Layer Code
The code we will be using for the layer is the following simple snippet as shown below.
def unbiased_coder_test_layer_function():
print ('Hello this is a test from the Layer to see if it works')
This is simple just to show how we can import this function inside a Lambda function. This off course could be a whole set of library files and things you want to require inside your Lambda.
All code can also be found in the Github repo here.
Once the create button is clicked and you proceed into making the Lambda you should see a screen like the below confirming the creation of your new Layer.
If you would like to deploy your Lambda layer programmatically you can do so using AWS CDK. I have written a full article about this which you can find here:
How To Setup AWS Lambda Using AWS CDK Python
How To Zip Lambda Layer Code
Now before we proceed into making the zip file to upload to our lambda creation process we need to make the zip file as a requirement as it was needed above.
This bit is very important as to how you are going to package this so when the zip Lambda Layer is unpacked on the Lambda instance it can be found from your Python code.
The first thing you need to keep in mind is the PATH of your Lambda Layer and how you can import properly the Layer code into your Lambda.
There’s a trick here where if you put the Layer code in a directory called python and zip it this way then it will be automatically added to your path. If you do not wish to do this, don’t worry I will talk about how you can still get it to work.
In this case lets go ahead and create the zip file from the command line using the ZIP command. Offcourse you can use any GUI command you wish to do the same thing too as long as the python folder is the main directory inside the zip and all your Python code would exist in there.
$ zip layer.zip python/layer.py adding: python/layer.py (deflated 15%)
This would have successfully created a zip file as shown above. To ensure it has preserved the format we need we can go ahead and unpack it to make sure that when it gets uploaded to the Lambda instance it’s associated with it will have the python directory as the root.
$ unzip layer.zip Archive: layer.zip inflating: python/layer.py $ ls -al -rw-r--r-- 1 user staff 274 Mar 30 20:08 layer.zip drwxr-xr-x 3 user staff 96 Mar 30 20:08 python $ ls -al python -rw-r--r-- 1 user staff 111 Mar 28 19:28 layer.py
The format of our zip structure seems correct. We will demonstrate later how we can use and associate this Layer with the Lambda code.
How Do I Import a Python Lambda Layer
One of the most common issues is that if you do not use the python directory as the root of your lambda you will need to do some extra code inside your lambda code in order for it to find your libraries.
To do this we simply going to leverage the sys.path that python maintains and uses to import Python libraries into your code. Since the Lambda does not know where your Layer code will live after unpacking you need to tell it where and give it a hint so it can work.
To do this we can simply use the following Python code at the beginning (prior the imports) and everything then will work.
import sys
sys.path.append('ROOT_ZIP_DIRECTORY')
import custom_module
In the example above we are assuming the root directory inside our zip file is ROOT_ZIP_DIRECTORY. If you do this then everything should just work fine for your Lambda code. Note I recommend using the default python directory to avoid having this extra boiler plate code.
How To Find Lambda Layer Size
The Lambda layer size can be easily calculated by using the du command in the directory you will be packaging up and sending.
This can be seen below:
$ du -h layer.zip 4.0K layer.zip (compressed) $ du -h python 4.0K . (uncompressed)
Basically once your Layer code is unpacked this would be the total size of your files. Note here that the layer code counts towards the total size of your Lambda so the same limitations will apply. If you would like more about the Lambda quotas you can check the AWS article here.
How Do You Add/Associate A Layer in Lambda
Now that we have the Lambda Layer fully created and deployed we can start associating with some of our Lambda functions.
The first thing you need to do is navigate to the Layer you want to add it too in AWS and select the Layer button as shown below.
Click on the Layers button as shown above with the red arrow and now you should get a prompt that allows you to associate your Lambda Layer with the Lambda function.
Go ahead and select custom layers as shown above and from the drop down you should see the newly created layer populate for you. Go ahead and select it and also pick the version you want to use for it. In my case I uploaded two different iterations of the Layer Code so I’m selecting version 2. If you are doing this for the first time you should see number 1 there instead.
Click Add once you finish and you should be all set with the Layer being associated with your Lambda function.
How To Import Layer In Lambda Function
Now that we have done the association with our Lambda we need to go ahead and check what code we need to use to import it.
In order to do this we will be using a very simple Python sample code that simple imports the function we defined earlier in our Lambda Layer. This can be seen below in the Python code.
# layer is located in 'python' directory
import json
from layer import unbiased_coder_test_layer_function
def lambda_handler(event, context):
unbiased_coder_test_layer_function()
As you can see we did not have to do the import magic here with the sys path as we used the method described earlier as keeping python the root directory in our zip layer structure. As before the code for all of this can be found in the Github repo here.
How To Test Layer In Lambda Function
To do the final test we will go ahead and execute our lambda function to see if it will print out the message that was defined earlier in our Lambda Layer function.
In order to do this we need to create an empty test event if you haven’t in your AWS console for this lambda function that we are using to test. Do note I have a full article on how to create lambda functions in AWS which you can find here:
How to Setup an AWS Lambda Python Function From Scratch
Once the empty event is created lets go ahead and do the execution and see what output our Lambda function will show.
As seen above the output we had in our layer code earlier (Hello this is ..) is being displayed.
This test successfully demonstrates how we used the Layer code in our Lambda function.
How To Add Python Requirements/Packages In AWS Lambda Layer
The process of doing this is identical to including our code in Python. Since packages are also basically python libraries you can do this by creating a virtual environment directory and install anything you want and simply doing a ZIP file with the contents of the requirements you have.
In order to do this you will need to run the virtualenv command and the start installing the packages. This can be seen below.
$ virtualenv python_packages
created virtual environment CPython3.9.10.final.0-64 in 272ms
creator CPython3Posix(dest=/Users/alex/code/unbiased/python-create-lambda-layer/python_packages, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/alex/Library/Application Support/virtualenv)
added seed packages: pip==22.0.3, setuptools==60.9.3, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source python_packages/bin/activate
$ pip install requests
Collecting requests
Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.1/63.1 KB 1.1 MB/s eta 0:00:00
....
Successfully installed certifi-2021.10.8 charset-normalizer-2.0.12 idna-3.3 requests-2.27.1 urllib3-1.26.9
$ deactivate
$ ls -al python_packages/lib/python3.9/site-packages
total 32
...
drwxr-xr-x 17 alex staff 544 Mar 30 20:37 urllib3
drwxr-xr-x 8 alex staff 256 Mar 30 20:37 urllib3-1.26.9.dist-info
drwxr-xr-x 12 alex staff 384 Mar 30 20:37 wheel
drwxr-xr-x 9 alex staff 288 Mar 30 20:37 wheel-0.37.1.dist-info
-rw-r--r-- 1 alex staff 0 Mar 30 20:37 wheel-0.37.1.virtualenv
In the example above I install the requests library. All you have to do next is zip the python3.9 folder and you should be set. Make sure you use the structure mentioned earlier if you do not want to go the extra step of setting up paths.
Can You Attach More Than One Lambda Layers In The Same Lambda Function
At the time of this writing you can add up to 5 layers in your AWS Lambda. Keep in mind that all of those add to the total allowed limit of your size of the entire deployment.
So adding more Layers will not increase that overall limit it will merely allow you to segregate your code better.
It’s good practise to only split up in multiple layers if there is a business use case.
For example lets say you have some database helper and an filesystem helper utility in your code, in that case one of those layers can be used in one lambda but both could be used in other lambdas. In such a case the situation justifies to split them up to keep things easier to debug.
Furthermore splitting your code in more layers can reduce the overall deployment size of your Lambda deployment because you would be only importing what you need rather than having one big library chunk that has everything.
Furthermore the import dependencies being loaded by multiple Layers can decrease execution speed so if you have split things up better your lambda will end up being faster in the long run and leaner in terms of memory consumption.
How to Deploy a Python Layer in AWS Lambda
You can deploy your Layer with one of the many deployment pipelines that exist such as:
- Jenkins
- AWS command line tool
- Serverless
- Zappa
I have a detailed article talking about these methods which you can find below.
How to deploy a Python Microservice in AWS
Conclusion
We were able to successfully go over How To Create An AWS Python Lambda Layer, hopefully I answered any questions you may have had and helped you get started on your AWS Lambda layer project.
If you found this useful and you think it may have helped you please drop me a cheer below I would appreciate it.
If you have any questions, comments please post them below or send me a note on my twitter. I check periodically and try to answer them in the priority they come in. Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.
Do you use Lambda Layers in your project?
I personally use Lambda Layers in all of my projects as they provide a clean and easy way to include re-useable code without having to paste it across all my AWS Lambda functions.
If you would like to visit the official AWS Lambda Layer documentation you can find a reference here.
If you would like to find more AWS related articles below: