How to Setup an AWS Lambda Python Function From Scratch

Introduction

Today I’m going to walk you through on how to setup an AWS Lambda Python Function From Scratch. We are going to do everything step by step starting from setting up your environment and any dependencies needed to getting a full working example. In the process I’m going to show you how this looks from the AWS console all the way to how the code can be adapted to be efficient. This simple task has small little tricks that you need to master when it comes to specifying paths and other limitations which we will demonstrate below and how you can address them to accomplish what you want.

I have been working in the Software industry for over 23 years now and I have been a software architect, manager, developer and engineer. I am a machine learning and crypto enthusiast with emphasis in security. I have experience in various industries such as entertainment, broadcasting, healthcare, security, education, retail and finance. I have been using AWS since inception and I’m familiar with all the technologies it has. My experience is not limited to the architecture but also doing a lot of hands on code and understanding the limits of each product AWS offers.

How to setup the environment for an AWS Lambda Python function

We are going to begin on setting up our environment in particular installing any dependencies and packages necessary. I assume you already have Python 3 installed and running in your system. If you haven’t you can check the official website of Python to get that installed before proceeding forward. Furthermore I’m assuming you already have access to an AWS Lambda account and own an SDK key. If you do not you can sign up for free with Amazon here to get started. I’m also making the assumption you have the Python Package manager pip. If you haven’t you can look into this guide to learn more about installing it.

Now that we have the basic requirements out of the way we can dive in and start setting up the system. Also I want to note that all of the code you will find in this guide can be found in github here.

How to create a virtual environment for an AWS Lambda Python function

• First install the virtual env using the python command: ‘pip install virtualenv’
• Then create a new virtual environment
• Finally you need to activate your virtual environment so we can start installing packages, please see below

main alex@DYNAMH ~/code/unbiased-coder/python-aws-lambda-guide > virtualenv env
created virtual environment CPython3.8.6.final.0-64 in 3773ms
creator CPython3Posix(dest=/home/alex/code/unbiased-coder/python-boto3-s3-guide/env, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/alex/.local/share/virtualenv)
added seed packages: pip==21.2.4, setuptools==57.4.0, wheel==0.37.0
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
main alex@DYNAMH ~/code/unbiased-coder/python-aws-lambda-guide > source env/bin/activate

main (env) alex@DYNAMH ~/code/unbiased-coder/python-aws-lambda-guide > pip install boto3 python-dotenv
Collecting boto3
Downloading boto3-1.18.46-py3-none-any.whl (131 kB)
|████████████████████████████████| 131 kB 1.1 MB/s
Collecting s3transfer<0.6.0,>=0.5.0
Downloading s3transfer-0.5.0-py3-none-any.whl (79 kB)
|████████████████████████████████| 79 kB 2.8 MB/s
Collecting botocore<1.22.0,>=1.21.46
Downloading botocore-1.21.46.tar.gz (8.2 MB)
|████████████████████████████████| 8.2 MB 11.5 MB/s
Collecting jmespath<1.0.0,>=0.7.1
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
|████████████████████████████████| 247 kB 10.1 MB/s
Collecting urllib3<1.27,>=1.25.4
Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
|████████████████████████████████| 138 kB 9.9 MB/s
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: botocore
Building wheel for botocore (setup.py) ... done
Created wheel for botocore: filename=botocore-1.21.46-py3-none-any.whl size=7933638 sha256=ee2d8a7f5bd91a7d2711b529706902a4a2a8fba97e69493757a8d1d461296d69
Stored in directory: /home/alex/.cache/pip/wheels/db/2a/b6/37624d07c0d7572bff3d08bd4bfd2c94b121f693278cd1ae77
Successfully built botocore
Collecting python-dotenv
Downloading python_dotenv-0.19.0-py2.py3-none-any.whl (17 kB)
Installing collected packages: six, urllib3, python-dateutil, jmespath, botocore, s3transfer, boto3, python-dotenv
Successfully installed boto3-1.18.46 botocore-1.21.46 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.0 six-1.16.0 urllib3-1.26.7 python-dotenv-0.19.0

main ✚ (env) alex@DYNAMH ~/code/unbiased-coder/python-boto3-s3-guide > python
Python 3.8.6 (default, Oct 23 2020, 14:59:35)
[GCC 9.3.0] on msys
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.18.46'
>>> import dotenv
>>> quit()

As you can see above the boto3 library got loaded successfully and the version is 1.18.46. This is as of Late 2021 so this may be different in your system based on when you install it.

How to setup AWS environment for an AWS Lambda Python function

Now that we have our bucket created we need to proceed further into setting up a way to interact with it programmatically. In order to do this Amazon AWS uses something called SDK keys. Those are necessary for the platform to know you are authorized to perform actions in a programmatic way rather than logging in the web interface and accessing the features via the console. So our next task is to find where and how those keys are configured and what is needed to set them up on our local computer to start talking to Amazon AWS Lambda.

How to add an AWS user 

If you do not have a user setup with AWS Lambda full permissions then I will walk you through on how to get this done in a simple step by step guide. The first thing you need to do is head over to the IAM page on AWS console and click on creating a new user.

In the next steps you can use the defaults except the part that is asking you to set the permissions. In this tab you want to expand below and type in the search lambda. Once you do that a bunch of permissions will be loaded for you to select from, for now you can simply select the Full permissions for Lambda as shown in the screenshot below.

You can skip the tags and proceed to add the user, the final screen summary should look like this.

The final confirmation screen should show you the access key and the secret key. You want to save those for your reference as we would be using them in our code later. This screen looks something like this:

Do note I redacted my access and secret key from the screenshot for obvious reasons but you should have them if everything worked successfully.

How to setup AWS secret, API key and region to access AWS Lambda Python function

Now that we have an access and secret key and our environment setup we can start writing some code. Before we jump into writing code that downloads uploads and lists files from our AWS bucket we need to write a simple wrapper that will be re-used across our applications that does some boiler plate code for the boto3 library.

One thing to understand here is that AWS uses sessions. Similar to when you login to the web console a session is initiated with a cookie and everything in a similar way this can be done programmatically. So the first thing we need to do before we start accessing any resource in our AWS environment is to start and setup our session. In order to do that we will leverage the library we installed earlier called dotenv. The reason we will use this is to access our secret and access key from the environment file. We use an environment file for security reasons such as avoiding to hardcode any values in our code base.

The environment file basically tells Python that the data will live in the process environment which is in memory and does not touch any source file. In a way this is similar to setting environment variables in your terminal but for convenience we set them in our .env file. The format of this would look something like this:

AWS_LAMBDA_ACCESS_KEY=AKIAQXXXXXXXXXFV6YYY
AWS_LAMBDA_SECRET_KEY=faaaabhjhjhdsjhdsjhdssU1
AWS_REGION=us-east-1

The data values above have been randomized for obvious reasons. But as you can see we are setting two variables here one for our access and one for our secret key which our code will be reading from in order to use them to initialize our AWS session. If we take a look at the boiler plate code it’s pretty simple and as described above it basically gives us an AWS session object to use later. Please don’t forget to change the region above according to your setup. This can be seen in the code below:

import os
import boto3
from dotenv import load_dotenv

def get_aws_keys():
    load_dotenv()
    return os.getenv('AWS_LAMBDA_ACCESS_KEY'), os.getenv('AWS_LAMBDA_SECRET_KEY')

def init_aws_session():
    access_key, secret_key = get_aws_keys()
    return boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name=os.getenv('AWS_REGION'))

How to create a new AWS Lambda function using console

Initially we will talk about how to create a new AWS Lambda function using console. Now that we have our environment setup and the initialization code base ready to go we can create our first lambda function in the AWS web console. Login in and quickly navigate to the AWS Lambda home page which can be found here. The default location of this will take you to US-EAST region feel free to change that if you want your test lambda to be somewhere else.

Go ahead and click on create new lambda function on the top right of the panel and you would be presented with a dialog on completing some details. If you are following this guide you can using the same naming as I did as it will be compatible with the github code found here that I have created and contains all code.

You need to complete threefields here:

• Create from scratch
• Select python interpreter as Python 3.8
• Name it unbiased_coder_lambda as that’s how the code will be referenced. If you decide to pick a different name and you are following the github repository then you need to adjust accordingly.

Once this information is completed please go ahead and click create to start provisioning your new AWS lambda function. If everything goes well you should see a confirmation of successful creation.

Now that we have the environment setup properly it’s time to start doing some coding and to test that our function is working.

How to make an example AWS Lambda function in Python

We will focus here on how to make an example AWS Lambda function in Python. Now that the lambda function has been successfully created AWS should have created for you some sample code that is ready to execute. Let’s go over some basics first and explain what a lambda is and where the output of your lambda is going and then we will slowly build our hello world example.

All lambda functions take what is called as an Event. An event is simply a JSON file that contains whatever arguments you want to execute your lambda as. For example in our case we will be passing to our lambda an event that contains a first and last name which we will be later be printing out inside our lambda function. The test event would simply look like this:

{
    "first_name" : "Unbiased",
    "last_name"  : "Coder"
}

Now that we have the event setup we can go ahead and click create test event from the drop down as shown below to make a new one and input the values as shown above.

Go ahead and click create and now your test drop down button should have a new event called unbiased-coder-test-event. Through out when testing this lambda from the console we will be using this test event to run our lambda code.

How to create an AWS Lambda using Boto3

The first part we need to do is talk about how to create an AWS Lambda using Boto3. The lambda function we will be creating would look like this:

import json

def lambda_handler(event, context):
    first_name = event['first_name']
    last_name  = event['last_name']
    
    return {
        'statusCode': 200,
        'body': json.dumps('Hello %s %s'%(first_name, last_name))
    }

What this does is simply return a 200 HTTP response with the first and last name we receive from the test event we created. One important thing to do here is to click on the deploy button to push your code and make it live. If you were to run this the output would look like this:

Test Event Name
unbiased-coder-test-event
Response
{
  "statusCode": 200,
  "body": "\"Hello Unbiased Coder\""
}
Function Logs
START RequestId: 543f0779-bc65-45c9-854e-de0a0bcc2ee0 Version: $LATEST
END RequestId: 543f0779-bc65-45c9-854e-de0a0bcc2ee0
REPORT RequestId: 543f0779-bc65-45c9-854e-de0a0bcc2ee0	Duration: 1.36 ms	Billed Duration: 2 ms	Memory Size: 128 MB	Max Memory Used: 50 MB	Init Duration: 140.12 ms

Request ID
543f0779-bc65-45c9-854e-de0a0bcc2ee0

As you can se above it’s returning Hello Unbiased Coder which is what we are expecting from it. This output can be found in both the execution window in the lambda but also in your cloudwatch logs.

To keep it brief if you are not familiar with cloudwatch it’s the default logging solution that Amazon provides to users. Basically any output of our Lambdas it’s configured to go in a log group under lambda functions and within that log group you can see the output of your functions.

And if you click on the log group you should be able to see the same output as the execution window we had before. Since this lambda does not have any print items it will not show anything there. If you would like to do that you can simply add print statements or use logger to output what you need.

How to launch an AWS Lambda Using Boto3

Moving on we will discuss how to launch an AWS Lambda using Boto3. Now that we have covered how you can run the Lambda function from the web console we will cover next how you can do this programmatically as it will make your life testing a lot easier and open up more capabilities such as debugging and stepping through the code which we will discuss later.

The first thing we need to do is write some code that will invoke our Lambda. This code will be leveraging our initializer code that we described above in the setup. Everything can also be found in the github repository here.

import boto3_helper

session = boto3_helper.init_aws_session()
lambda_client = session.client('lambda')

lambda_payload = '{"first_name" : "Unbiased", "last_name"  : "Coder"}'

res = lambda_client.invoke(FunctionName='unbiased_coder_lambda', InvocationType='RequestResponse', Payload=lambda_payload)

print('Lambda return: ', end='')
while True:
    try:
        content = res['Payload'].next()
        print(content)
    except StopIteration:
        break

As it can be seen in the code above we are using the same Test event payload we had setup previously in the web console of AWS. The new things we added are the following:

• Invocation of lambda code using the boto3 session, this is a synchronous invocation (you can invoke it asynchronously too if you adjust the parameter of Invocation type)
• Then we loop using the Payload iterator from boto3 to get the full response.

If we were to execute this code in our terminal we will get the following output:

main (venv) alex@DYNAMH ~/code/unbiased-coder/python-aws-lambda-guide > python ./invoke_lambda.py
Lambda return: b'{"statusCode": 200, "body": "\\"Hello Unbiased Coder\\""}'

The status code is what our lambda was sending which is correct and furthermore we can also see our return body code of ‘Hello Unbiased Coder’. This is similar to what we had seen earlier when we invoked the lambda from the Amazon web console so we know our code works.

How to access environment variables in AWS Lambda Python

One of the things you will need to know is how to access environment variables in AWS Lambda Python. In order to access environment variables in AWS lambda python first we need to add a few to test it. If you go into your lambda configuration section there’s something called environment variables in the left hand side bar. Click on it and the below screen should be visible:

If you click edit you can start adding some of them, for example in our hello world lambda we will add one called COUNTRY which our lambda will further read and print it out in the response code. To do this just click edit and add the variable to whatever value you want. In our case we set it to USA and this can be seen below.

The next step is now to test our lambda by reading the environment variable and putting it in the response. Our code needs to be adjusted as follows:

country = os.environ['COUNTRY']
return {
   'statusCode': 200,
   'body': json.dumps('Hello I am %s %s and I am from %s'%(first_name, last_name, country))
}

If we invoke it using our Makefile which can be found in git we should see the updated output:

main (venv) alex@DYNAMH ~/code/unbiased-coder/python-aws-lambda-guide > make run
python ./invoke_lambda.py
Lambda return: b'{"statusCode": 200, "body": "\\"Hello I am Unbiased Coder and I am from USA\\""}'

If you are looking for a more secure approach on this you can also look into Amazon KMS which allows you to add encryption and decryption of your environment variables. This is a good alternative if your lambda is using sensitive information that needs to be exposed in the environment for initialization reasons.

How to create zip file for AWS Lambda Python and update the code

The final step is to how to create zip file for AWS Lambda Python and update the code. In order to create the zip file we can use the zip command and include our updated source file or even directory. This would enable us to update our lambda code. Lets walk through the process of what it takes to programmatically update your lambda code without having to login to the web console using the aws command line tool.

If you haven’t already installed the AWS command line tool you can find it here. The process of configuring it is pretty straight forward, once installed you need to invoke the configure command of it as shown below:

main (venv) alex@DYNAMH ~/code/unbiased-coder/python-aws-lambda-guide > aws configure
AWS Access Key ID [None]: YOURACCESSKEY
AWS Secret Access Key [None]: YOURSECRETKEY
Default region name [us-east-1]: us-east-1
Default output format [None]:

Now that we have our aws command line setup we can go ahead and issue the two commands to archive the updated code and upload it.

main (venv) alex@DYNAMH ~/code/unbiased-coder/python-aws-lambda-guide > zip unbiased-coder-lambda.zip test_lambda.py
updating: test_lambda.py (deflated 37%)
main (venv) alex@DYNAMH ~/code/unbiased-coder/python-aws-lambda-guide > aws lambda update-function-code --function-name unbiased_coder_lambda --zip-file fileb://unbiased-coder-lambda.zip
{
"FunctionName": "unbiased_coder_lambda",
"FunctionArn": "arn:aws:lambda:us-east-1:033533902081:function:unbiased_coder_lambda",
"Runtime": "python3.8",
"Role": "arn:aws:iam::033533902081:role/service-role/unbiased_coder_lambda-role-wr5wayco",
"Handler": "lambda_function.lambda_handler",
"CodeSize": 334,
"Description": "",
"Timeout": 3,
"MemorySize": 128,
"LastModified": "2021-09-23T20:03:28.282+0000",
"CodeSha256": "WZTOtPXjd7IbdFHkwcFZ9xmLlkzyFZI5aoPkba9q0gs=",
"Version": "$LATEST",
"TracingConfig": {
"Mode": "PassThrough"
},
"RevisionId": "1c1a55c7-7d17-4b79-9971-6fa94c97c453",
"State": "Active",
"LastUpdateStatus": "Successful",
"PackageType": "Zip"
}

As shown above the command to issue the update is called ‘update-function-code’ which is part of the lambda component of the aws cli, and to zip the file we simply issue a zip command containing our test hello world lambda. The result of the command should indicate the status in the LastUpdatedStatus and it should be Successful. To simplify the process I created a Makefile that will upload our code to the AWS environment:

upload:
	echo "Preparing zip file for upload..."
	zip unbiased-coder-lambda.zip test_lambda.py
	echo "Updating Lambda code"
	aws lambda update-function-code --function-name unbiased_coder_lambda --zip-file fileb://unbiased-coder-lambda.zip
	aws lambda update-function-configuration --function-name unbiased_coder_lambda --handler test_lambda.lambda_handler
run:
	python ./invoke_lambda.py

This lets us upload our lambda in AWS just by running make upload. The second one updates the configuration because by default the update-function-code changes the handler name and we need to set it back to what ours is.

Conclusion

Overall I’m a big fan of doing things programmatically when it comes to AWS and I believe Amazon has provided a great library with a lot of flexibility but it does have it’s limitations as demonstrated above. The bigger problem here is that we need to be aware of those and be adaptable after all nothing in life is perfect! If you enjoyed reading How to Setup an AWS Lambda Python Function From Scratch I would appreciate if you can drop me a line below to cheer me up.

If you have any questions, comments below I check periodically and try to answer them in the priority they come in.
Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.

I include some articles that I wrote on Boto3 and AWS services below that you may find useful.

• How To Create An AWS Python Lambda Layer
• How To Setup AWS Lambda Using AWS CDK Python
• AWS Lambda vs AWS Step Functions
• How To Store Global Environment Variables In AWS
• How To Detect If I am Running In AWS Lambda Environment

Which AWS service do you use programmatically?