Introduction
Today we will discuss about the limitations of AWS lambda functions. I have been using the AWS ecosystem in various projects since the introduction of lambda functions and I’ve seen the different growth stages it has gone through. While I’m personally not a big fan of task based approach I do see merit in some cases where a lambda can be useful for a team. In this article I’m only going to talk about the limitations and shortfalls of lambdas.
I have been working in the Software industry for over 23 years now and I have been a software architect, manager, developer and engineer.
I am a machine learning and crypto enthusiast with emphasis in security. I have experience in various industries such as entertainment, broadcasting,
healthcare, security, education, retail and finance.
What is the AWS Lambda Function time limit
The idea of a lambda is to break down your code into smaller execution tasks hoping that the break down will be lower than the upper limit of the execution time which is currently set too as 15 minutes. While this is fine for most cases it gets problematic when you are dealing with stochastic/explorative tasks. Because I want to always reference some real life examples I’ll give you an easy one to understand.
Consider the case where you need execute a query on a big DynamoDB table and you are looking for a record. The index is decided, the query is sent out but the query takes longer than 15minutes to complete.
How? Inheriting a badly organized table could be one reason, the code using scan instead of query to fetch the results a badly selected index etc. The issue here is that a code that would probably work 90% of the time for some queries it may expire and run out of execution and this really is a problem when it is tied to an API gateway call which has an even lower expectancy for a result which is 30 seconds!! So you are left out of options here since the execution time is a cap. You can try cascade or split up your query or even optimize it. However at this point you are doing things to optimize your code due to a technology limitation. This is not necessarily a bad thing your code should always be optimized and your data should always be well organized but what happens if you inherit a badly organized project and have a tight deadline to work with? Naturally you have to look elsewhere, I believe good alternatives for this is EC2 or dockerized AWS Batch jobs where the time limit isn’t there. You can also run simple ECS/EKS dockers to execute your task.
What is the AWS Lambda Function disk space limit
Similar to above I think the limit of having 512MB I believe is very small. Let me start by explaining a real life scenario where this could really be an issue and before you say why don’t you just use EC2 please think this through and try to understand that some bigger organizations do not want to maintain an EC2 instance nor do patch management due to cost facts. And yes I’m aware that EFS is a solution to this but it does not come configured like that out of the box nor it’s an option when you are creating a new lambda function so I don’t consider that part of the feature AWS is advertising but an add-on. So a good example is doing some kind of ETL job (extract – transform – load) and you are processing data. This data may end up requiring to be uploaded to S3 or temporarily saved on disk for some transformation. Unfortunately with the limit of the disk space you are now forced to use S3 multipart uploads and risking running out of execution time in your batch job. Why? Because you won’t be able to split your code otherwise and the batch job will need to handle both tasks as there’s no permanent storage (unless offcourse you use EFS). Another example could be a video or media transformation, furthermore you may need to load a big dataset that cannot be cached in memory (which has a 10GB limit). As you can tell the examples can keep going on I personally do not think there should be a limit in AWS lambdas other than the fact that Amazon did no spend the time to integrate it seemlessly with EFS or some other similar solution.
What is the AWS Lambda Function initialization latency
I’m going to discuss here about the time that it takes from the moment a lambda function is called to the point that the code actually gets executed. This is also referred to as initialization time of the lambda. There are a lot of things mentioned here about it for example people talk a lot about keeping a lambda hot which just means having it run without having to go through that initial latency time for the lambda to get setup. Just to offer a bit more context a lambda really is a server/docker container that gets created based on the parameters required by the user at the time of the creation. So Amazon needs sometime to spin up the instance, attach any layer code to it and start it up. This naturally could never be zero it will always take time even if the instance in terms of hardware is always hot there’s time needed to copy over potential layers and the code to execute along with the runtime (python, js etc.)
I think this topic of keeping a lambda hot has been over analyzed and discussed in the past, I don’t really want to go into detail about it as there are a lot of great online references even by Amazon and other third parties, you can find one below that discusses this.
https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/
The idea here is that if you require your code to execute with low latency this is unlikely to always happen and it boils down to how often your lambda is being invoked as an API request. What this means in a real life situation? Your app, game, frontend etc may be expecting a result in a timely manner and it simply won’t get it because the lambda may have a longer than expected initialization time to be ready to execute the code we have written. This is a normal process and it saves you money since the instance is not always running. If you really care to keep your lambda hot you need to do a hack and make some pseudo ping or other methods discussed online. If your lambda gets invoked very often and doesn’t have a ‘sleep’ period then it may always be hot.
Conclusion
I generally think AWS lambdas are a great technology but they serve their purpose and they are not always applicable. Sometimes the memory and processing restraints we experience may set us back in a bigger way than first imagined. So it always makes sense to go over your project requirements and scope properly what you really need and if Lambdas are really a fit for your use-case. If not there’s other technologies such as Amazon EC2 which could also solve your problem but may require a little more maintenance and work. I talk about these in my AWS articles which you could find more about in this section.
Today I discussed about some limitations of AWS lambda functions, having said that do not let that deter you from using them, it’s a great technology and fits certain needs!
If you found this article useful and you think it may have helped you please drop me a cheer below I would appreciate it.
If you have any questions, comments below I check periodically and try to answer them in the priority they come in.
Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.
If you would like to learn more about AWS please take a look at my section here for more articles.
Do you use AWS Lambda functions?
You can find some articles here related to AWS Lambda:
- AWS Lambda vs AWS Step Functions
- How to scale Python in AWS Lambda
- How To Create An AWS Python Lambda Layer
- How to Setup an AWS Lambda Python Function From Scratch
- How To Detect If I am Running In AWS Lambda Environment