How To Scale Python in AWS Lambda

Introduction

Today we will discuss how to scale Python in AWS Lambda. I have been using a lot of Python lately to implement ETL (extract transform load) jobs along side with DynamoDB as the database of choice. While these tasks are always invoked with small jobs there’s some that does take longer than usual to complete. Considering the limitations of an AWS Lambda function as discussed in a previous article I wrote, there’s some things you can do to mitigate this and have your code functional and completed in a timely manner. I will try to break down this post into things you can do so your code doesn’t hang there and reach the 15minute execution limit.

I have been working in the Software industry for over 23 years now and I have been a software architect, manager, developer and engineer. I am a machine learning and crypto enthusiast with emphasis in security. I have experience in various industries such as entertainment, broadcasting, healthcare, security, education, retail and finance.

How to scale Python in AWS Lambda
How to scale Python in AWS Lambda

Below I will talk about two methods:

  • Threading
  • Multiprocessing

Both of these approaches have positives and negatives, however we will try to focus more on the scaling aspect of it and address the issues at another time. The idea here is to try to make our Lambda function finish execution as fast as possible.

How to use threading in Python AWS Lambda

As per usual threading we know that if you have a task that’s taking too long to complete and is usually awaiting for something like a network request or a file to be changed then there’s things you can do to avoid this I/O blocker. Threading is the easiest solution in this case as it lets you execute something else in parallel. There’s two ways you can do this either have a pool. As per usual threading we know that if you have a task that’s taking too long to complete and is usually awaiting for something like a network request or a file to be changed then there’s things you can do to avoid this I/O blocker. Threading is the easiest solution in this case as it lets you execute something else in parallel. There’s two ways you can do this either have a pool manager:

https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.ThreadPool

or you can use some synchronization with .join to let your threads communicate. Lets consider a simple use-case here where we are pulling records from the database and do something with them either adding them in a file or transforming them into other data and inserting them again into the database. This kind of transformation will typically take very long to complete as you will be basically pulling 1 record at a time and processing it. Instead you can pull a limit of lets say 2048 records at a time start a thread and let them run in the background. This may hammer a bit your database but if you are using something that runs in a cloud ecosystem like DynamoDB or Redshift this should not be an issue. By doing this you are making your code concurrent and can see massive increases in execution time. So a task that could take 15mins will now probably take less than 1 minute if it was I/O blocked like the example we just discussed.

Python threading - AWS Lambda
Python threading – AWS Lambda

How to use multiprocessing in Python AWS Lambda

As per the example we gave above the same thing can be applied using multi-processing. The idea here is that your code will start different processes instead of threads to run the execution of the tasks. Please keep in mind that usually Lambdas are limited on CPU resources so I advice against scaling this too much and relying more on a hybrid approach of multi-threading and a few process instances. As time passes your lambda will scale better and better based on the resources that it has been allocated at the time of execution.

https://docs.python.org/3/library/multiprocessing.html#the-process-class

One thing to consider here is that when a new process is created a stack copy is made and inherited so the footprint of making a new process is much heavier than creating a new thread. Taking this into account your memory needs in your AWS Lambda function will be greater than if you were to use a multi-threaded approach. If cost is an issue you may want to prefer threading over processes. If this is not an issue and you want the absolute scalability and speed combined with a serverless environment Multiprocessing is a better choice for you.

Python multiprocessing - AWS Lambda
Python multiprocessing – AWS Lambda

Conclusion

I tried to outline two popular methods to speed up your python code in general but they also apply very well in the AWS Lambda environment. These solutions I have used in the past successfully and have managed to port lambdas that never finished and always hit the maximum timeout limit in a way that they execute super fast. One of the biggest reasons is that Python has the global interpreter lock which gets in the way of it’s performance among other things and often waits on I/O blockage. We bypass this using threading and multiprocessing.

As mentioned above the choice entirely depends on your budget, needs and what you are trying to accomplish. I personally find myself surfacing and getting the job done simply with a multi-threaded environment. I never had major speed improvements by using a multi-processing environment. This may not be the case for you especially if you are in the field of editing or video processing where your AWS Lambda is doing a lot of different transformations that are timely and have CPU blocking.

Do you have any other ideas on how to improve execution time and be within the lambda limits? Please put your comments below I would be really interested to hear them. Also I would appreciate any feedback/corrections on what I wrote.

If you enjoyed this article you can find some similar ones here:

Leave a Comment

Your email address will not be published. Required fields are marked *