Introduction
Today we will discuss on everything you need to know about Python Boto3 API Gateway: Post, Get, Lambda/EC2, Models, Auth in simple and easy to follow guide with lots of hands-on examples.
Would you like to know to programmatically instrument API Gateway?
You came to the right resource, all code, examples can be found in the GIT hub repository I created for this article here.
More specifically we will discuss:
- Listing the existing API Gateway configuration and api end points programmatically
- Invoking and Testing API Gateway end points programmatically
- Adding Authentication to our Boto3 API Gateway requests
I have been working in the Software industry for over 23 years now and I have been a software architect, manager, developer and engineer. I am a machine learning and crypto enthusiast with emphasis in security. I have experience in various industries such as entertainment, broadcasting, healthcare, security, education, retail and finance.
How to install Boto3 to connect to use API Gateway
We are going to begin on setting up our environment in particular installing any dependencies and packages necessary. I assume you already have Python 3 installed and running in your system. If you haven’t you can check the official website of Python to get that installed before proceeding forward. Furthermore I’m assuming you already have access to an AWS EC2 account and own an SDK key. If you do not you can sign up for free with Amazon here to get started. I’m also making the assumption you have the Python Package manager pip. If you haven’t you can look into this guide to learn more about installing it.
Now that we have the basic requirements out of the way we can dive in and start setting up the system. Also I want to note that all of the code you will find in this guide can be found in github here.
How to Create a Python Virtual Environment for Boto3 API Gateway
- First install the virtual env using the python command: ‘pip install virtualenv’
- Then create a new virtual environment
- Finally you need to activate your virtual environment so we can start installing packages, please see below
main alex@DYNAMH ~/code/unbiased-coder/python-boto3-pagination-collections > virtualenv env
created virtual environment CPython3.8.6.final.0-64 in 3773ms
creator CPython3Posix(dest=/home/alex/code/unbiased-coder/python-boto3-ec2-guide/env, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/alex/.local/share/virtualenv)
added seed packages: pip==21.2.4, setuptools==57.4.0, wheel==0.37.0
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
main alex@DYNAMH ~/code/unbiased-coder/python-boto3-pagination-collections > source env/bin/activateHow to Install pip dependencies for Boto3 API Gateway
Next we need to go ahead and install the Python dependencies to be able to use the boto3 library. You can do this by running the pip tool as shown below. Keep in mind make sure your virtual environment is activated before you run this step. If you wish to use it without having a virtual environment which I do not recommend you can go ahead and simply install it globally in your user account.
main (env) alex@DYNAMH ~/code/unbiased-coder/python-boto3-pagination-collections > pip install boto3 python-dotenv requests
Collecting boto3
Downloading boto3-1.18.46-py3-none-any.whl (131 kB)
|████████████████████████████████| 131 kB 1.1 MB/s
Collecting s3transfer<0.6.0,>=0.5.0
Downloading s3transfer-0.5.0-py3-none-any.whl (79 kB)
|████████████████████████████████| 79 kB 2.8 MB/s
Collecting botocore<1.22.0,>=1.21.46
Downloading botocore-1.21.46.tar.gz (8.2 MB)
|████████████████████████████████| 8.2 MB 11.5 MB/s
Collecting jmespath<1.0.0,>=0.7.1
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
|████████████████████████████████| 247 kB 10.1 MB/s
Collecting urllib3<1.27,>=1.25.4
Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
|████████████████████████████████| 138 kB 9.9 MB/s
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: botocore
Building wheel for botocore (setup.py) ... done
Created wheel for botocore: filename=botocore-1.21.46-py3-none-any.whl size=7933638 sha256=ee2d8a7f5bd91a7d2711b529706902a4a2a8fba97e69493757a8d1d461296d69
Stored in directory: /home/alex/.cache/pip/wheels/db/2a/b6/37624d07c0d7572bff3d08bd4bfd2c94b121f693278cd1ae77
Successfully built botocore
Collecting python-dotenv
Downloading python_dotenv-0.19.0-py2.py3-none-any.whl (17 kB)
Collecting requests
Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Installing collected packages: six, urllib3, python-dateutil, jmespath, botocore, s3transfer, boto3, python-dotenv
Successfully installed boto3-1.18.46 botocore-1.21.46 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.0 six-1.16.0 urllib3-1.26.7 python-dotenv-0.19.0 requests-2.26.0The two packages we installed are:
- boto3: This is the core Python AWS library we will be using in this guide
- dotenv: We will use this library to pass in sensitive information to it that we do not want to have hardcoded in our code such as the AWS credentials
- requests: Requests library that we will use later to test the API Gateway endpoint
Verifying it works
Now that we have setup our system we need to verify the library is installed properly and it works. You can do this by simply checking in a python shell using the following command shown below, if you encounter an error please delete your virtual environment and try again. If the problem still persists please drop me a line below and I will try to help you.
main ✚ (env) alex@DYNAMH ~/code/unbiased-coder/python-boto3-pagination-collections > python
Python 3.8.6 (default, Oct 23 2020, 14:59:35)
[GCC 9.3.0] on msys
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.18.46'
>>> import dotenv
>>> quit()
>>> import requests
>>> requests.__version__
'2.22.0'As you can see above the boto3 library got loaded successfully and the version is 1.18.46. This is as of Late 2021 so this may be different in your system based on when you install it.
How to add an AWS user for Boto3 API Gateway
Before we start writing code and communicating with our API we need to get the API and Secret key that we will use alongside with the boto3 library that we previously setup. This is basically setting up permissions for the user to be able to manipulate and connect to the AWS infrastructure. First navigate to the IAM section and click add user. This will invoke the dialog as shown below.
The important part we need to select is the Access key – Programmatic access. This will allow us to programmatically control things using our code. The next important part is setting the correct permissions in order to access AWS.
How to select an AWS policy for API Gateway
You can go ahead and search for the apigateway policy you would want to control using Boto3. In this example we will choose the API Gateway Administrator entry as shown in the picture below. If you want to know more about controlling other AWS services and setting them up I have written a lot of articles that you can find the in my AWS section here.
So you can go ahead and click next and on the next dialog select the Attach existing policies directly. From this tab we need to select full permissions for API Gateway as discussed earlier.
How to get Boto3 API Gateway credentials
Now that we have selected the permissions to attach to our profile we can go ahead and add the user.
Once the user is created you should get a confirmation screen as seen below with two important parts:
- API Key: This is similar to your username which your code will be using to authenticate itself to the AWS service (in this case the IAM service)
- Secret Key: This is similar to the password which will be used to give you the permissions necessary to access the AWS service
The confirmation screen below shows how this would look like.
Once important thing to note here is that the Secret key is not shown but you will need to expand and save it now as we will be using it later in the document when we talk on how to setup an AWS Session.
How to Setup API Gateway
I have written an extensive guide which you could find below on how to do this. This article builds upon that API we have created:
Create Rest endpoint using API Gateway
How to get API Gateway Version, rate limit and throttle settings using Boto3
First we are going to demonstrate with code using Boto3 how to get the following information:
- API Gateway Version
- Rate limits and thresholds
- Throttle settings
We are leveraging the boto3 helper that has been previously discussed in detail here. Basically that module lets us setup a session with the AWS infrastructure and connect to it.
The next thing we need to do is acquire an api gateway object using the client which is the lower level AWS interface. In this case we need to use that instead of a resource to get access to a wider variety of functionality on on doing queries on our API Gateway Service.
import boto3_helper
import pprint
session = boto3_helper.init_aws_session()
api_gateway = session.client('apigateway')
pprint.pprint (api_gateway.get_account())
The important function above is called get_account. This gives us all the high level information associated with the API Gateway setup of the system.
✘ main ● (venv) alex@DYNAMH ~/code/unbiased-coder/python-boto3-api-gateway-guide > python3 ./boto3_api_gateway_get_account.py
{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',
'content-length': '107',
'content-type': 'application/json',
'date': 'Mon, 01 Nov 2021 20:52:23 GMT',
'x-amz-apigw-id': 'IJH9QJyAoAMEdeQ=',
'x-amzn-requestid': 'eacff046-c3bd-44b6-94f4-de74fabca8a0'},
'HTTPStatusCode': 200,
'RequestId': 'eacff046-c3bd-44b6-94f4-de74fabca8a0',
'RetryAttempts': 0},
'apiKeyVersion': '4',
'features': ['UsagePlans'],
'throttleSettings': {'burstLimit': 5000, 'rateLimit': 10000.0}}
In the output above we can make the following observations:
- The API Gateway endpoint supports version 4
- There’s no useage plan setup or limits
- We have a 5000 burst limit
- And our rate limiter is set to 10000
How to get API Gateway API Keys using Boto3
Next one thing that is of importance is to retrieve information about any API Keys that are created to interact with the API Gateway endpoints.
A use case for the above would if you are trying to programmatically send an API Request and use the API Authentication approach. This lets us verify the client making the request to API Gateway.
import boto3_helper
import pprint
session = boto3_helper.init_aws_session()
api_gateway = session.client('apigateway')
pprint.pprint (api_gateway.get_api_keys(limit=5, includeValues=True))
When sending the retrieval function get_api_keys we specify a limit of 5 keys (you can set a higher value but this account only has one) and the important part is to include the Values of the keys which is the second keyword argument. Executing the code snipet above should give us the results that we can see below.
✘ main ● (venv) alex@DYNAMH ~/code/unbiased-coder/python-boto3-api-gateway-guide > python3 ./boto3_api_gateway_get_api_keys.py
{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',
'content-length': '192',
'content-type': 'application/json',
'date': 'Mon, 01 Nov 2021 20:53:26 GMT',
'x-amz-apigw-id': 'IJIHGK0LIAMEd8A=',
'x-amzn-requestid': 'a348f002-8411-4eb2-99b0-bb1089624076'},
'HTTPStatusCode': 200,
'RequestId': 'a348f002-8411-4eb2-99b0-bb1089624076',
'RetryAttempts': 0},
'items': [{'createdDate': datetime.datetime(2021, 11, 1, 16, 45, 52, tzinfo=tzlocal()),
'enabled': True,
'id': 'h1dghq91wk',
'lastUpdatedDate': datetime.datetime(2021, 11, 1, 16, 59, 18, tzinfo=tzlocal()),
'name': 'UnbiasedCoderKey',
'stageKeys': [],
'value': 'w96hFN96XM5ve3gBSaaa2mXxwjD53lEMQYLHWH00'}]}There’s a few important things here that this is returning to us, the results go in an items field because it’s a list of entries. For example there could be more than one API Gateway API Key associated with it. Each key has the following attributes:
- If it’s on or off this is signified by the enabled flag
- The ID it has, this is a simple identifier
- The name, this is exactly as we set it up earlier in this case Unbiased Coder Key
- And the most important field which is the key itself shown in the value element.
How to list API Gateway Models using Boto3
Moving on someone may want to dynamically list all the models that we have associated with the API Gateway endpoint. By leveraging the same boilerplate code helper that we did before we establish the session and then use the get_models function which give us the results. This function takes a parameter which cannot be skipped the Rest API ID which we have seen earlier when setting up our endpoint. We will also show later in this article how to get this dynamically.
import boto3_helper
import pprint
session = boto3_helper.init_aws_session()
api_gateway = session.client('apigateway')
pprint.pprint (api_gateway.get_models(restApiId='41a9v8ivkj'))Once we execute this code we should retrieve the full schema of all of the models we previously created on the AWS console. Note that all those can also be created programmatically using Boto3. A full reference of this can be cited here.
main ● (venv) alex@DYNAMH ~/code/unbiased-coder/python-boto3-api-gateway-guide > python3 ./boto3_api_gateway_get_models.py
{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',
'content-length': '1728',
'content-type': 'application/json',
'date': 'Mon, 01 Nov 2021 20:54:44 GMT',
'x-amz-apigw-id': 'IJITRLJaoAMEdnA=',
'x-amzn-requestid': 'f80a9ce7-f5cb-4568-b64f-2d9cc1db205a'},
'HTTPStatusCode': 200,
'RequestId': 'f80a9ce7-f5cb-4568-b64f-2d9cc1db205a',
'RetryAttempts': 0},
'items': [{'contentType': 'application/json',
'description': 'This is a default error schema model',
'id': 'fq2d7o',
'name': 'Error',
'schema': '{\n'
' "$schema" : '
'"http://json-schema.org/draft-04/schema#",\n'
' "title" : "Error Schema",\n'
' "type" : "object",\n'
' "properties" : {\n'
' "message" : { "type" : "string" }\n'
' }\n'
'}'},
{'contentType': 'application/json',
'id': 'lti2l8',
'name': 'PersonGetRequest',
'schema': '{\r\n'
' "$schema": '
'"http://json-schema.org/draft-04/schema#",\r\n'
' "title": "Person",\r\n'
' "type": "object",\r\n'
' "properties": {\r\n'
' "name": {\r\n'
' "type": "string"\r\n'
' },\r\n'
' "surname": {\r\n'
' "type": "string"\r\n'
' },\r\n'
' "age": {\r\n'
' "description": "Age in years",\r\n'
' "type": "integer",\r\n'
' "minimum": 21\r\n'
' }\r\n'
' },\r\n'
' "required": ["name", "surname", "age"]\r\n'
'}'},
....
....As expected we are seeing the two models we created along with all their configuration attributes and data type definitions. The output above has been concatenated for simplicity as it’s identical to what we have seen earlier when we were working on the AWS console.
How to get API Gateway Resource Methods and Path using Boto3
In a similar way to the models we can also retrieve the resource methods and path of our API Gateway that allow us to interact with our backend system. The method we will be using here is called get_resources and works exactly as the models method accepting as an argument the Rest API ID.
import boto3_helper
import pprint
session = boto3_helper.init_aws_session()
api_gateway = session.client('apigateway')
pprint.pprint (api_gateway.get_resources(restApiId='41a9v8ivkj'))If we were to execute the above code snipet we will get back all the resources we previously created.
main ● (venv) alex@DYNAMH ~/code/unbiased-coder/python-boto3-api-gateway-guide > python3 ./boto3_api_gateway_get_resources.py
{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',
'content-length': '158',
'content-type': 'application/json',
'date': 'Mon, 01 Nov 2021 20:55:15 GMT',
'x-amz-apigw-id': 'IJIYDKzpoAMEdeQ=',
'x-amzn-requestid': 'db431e3d-af35-48e0-80c9-f85441077d6c'},
'HTTPStatusCode': 200,
'RequestId': 'db431e3d-af35-48e0-80c9-f85441077d6c',
'RetryAttempts': 0},
'items': [{'id': '5mpowz65l9', 'path': '/'},
{'id': 'wb3ufm',
'parentId': '5mpowz65l9',
'path': '/person',
'pathPart': 'person',
'resourceMethods': {'GET': {}, 'POST': {}}}]}In this case we have defined a GET and a POST request both pointing to the /person path. One important thing to note is that every resource has it’s own unique identifier associated with it and this is not to be confused with our Rest API Identifier which we will demonstrate below on how to retrieve programmatically.
How to get API Gateway Deployments, API URL, Rest API ID and Name using Boto3
Now that we have the models and know what parameters our methods take, along with the URL locations we are missing one last bit and this is the base URL to make the requests too. API Gateway creates for us a staging environment which has all this information. In this staging environment one unique thing is the base URL for our methods. Knowing this base URL we can assemble full URLS along with the parameters that we have previously identified.
Getting the information for the API Gateway Deployment is a little more involved as it’s a two step process. First we need to get an export of our configuration in JSON format. This is done using the exportType type parameter shown below along with the staging environment name. In our case we deployed with a staging environment name of Dev. Finally we also need to give it the Rest API identifier to be used that’s associated with that deployment.
Once this is done then we need to use read for the result of the body to a JSON buffer that gets passed on to our standard Python JSON loads function. The important bit of that function that contains the base URL among other things is the servers attribute.
import boto3_helper
import pprint
import json
session = boto3_helper.init_aws_session()
api_gateway = session.client('apigateway')
pprint.pprint (api_gateway.get_rest_apis())
pprint.pprint (api_gateway.get_deployments(restApiId='41a9v8ivkj'))
export = api_gateway.get_export(restApiId='41a9v8ivkj', stageName='dev', exportType='oas30')['body']
print('URL: ', json.loads(export.read())['servers'])
Executing the above code should display among other information the base URL along with the staging environment name.
main ● (venv) alex@DYNAMH ~/code/unbiased-coder/python-boto3-api-gateway-guide > python3 ./boto3_api_gateway_get_restapi_id.py
{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',
'content-length': '200',
'content-type': 'application/json',
'date': 'Mon, 01 Nov 2021 21:07:50 GMT',
'x-amz-apigw-id': 'IJKOGK3KoAMEbnA=',
'x-amzn-requestid': '1a459655-8f60-42a2-b245-5af24325998a'},
'HTTPStatusCode': 200,
'RequestId': '1a459655-8f60-42a2-b245-5af24325998a',
'RetryAttempts': 0},
'items': [{'apiKeySource': 'HEADER',
'createdDate': datetime.datetime(2021, 10, 26, 21, 13, 46, tzinfo=tzlocal()),
'disableExecuteApiEndpoint': False,
'endpointConfiguration': {'types': ['REGIONAL']},
'id': '41a9v8ivkj',
'name': 'UnbiasedCoderAPI'}]}
{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',
'content-length': '113',
'content-type': 'application/json',
'date': 'Mon, 01 Nov 2021 21:07:51 GMT',
'x-amz-apigw-id': 'IJKOIJk2IAMEbdA=',
'x-amzn-requestid': '0acc739b-8f45-4dcf-bddf-b1c182585398'},
'HTTPStatusCode': 200,
'RequestId': '0acc739b-8f45-4dcf-bddf-b1c182585398',
'RetryAttempts': 0},
'items': [{'createdDate': datetime.datetime(2021, 11, 1, 16, 33, 44, tzinfo=tzlocal()),
'description': 'dev',
'id': '0l0dj4'},
{'createdDate': datetime.datetime(2021, 11, 1, 16, 46, 37, tzinfo=tzlocal()),
'id': 'kibx2i'}]}
URL: [{'url': 'https://41a9v8ivkj.execute-api.us-east-1.amazonaws.com/{basePath}', 'variables': {'basePath': {'default': '/dev'}}}]
As shown above the base URL for this case is: https://41a9v8ivkj.execute-api.us-east-1.amazonaws.com. However this alone is not sufficient as it requires to be followed by the staging environment this is taking place and in our example above this is Dev. So adding the resource of the person object we found above the full URL which we will query below to make requests to our API is:
https://41a9v8ivkj.execute-api.us-east-1.amazonaws.com/dev/person
Which accepts the POST and GET requests with the parameters specified in our pre-defined models. In the last section below we will be accessed from within an EC2/Lambda or even outside an AWS environment.
How to invoke API Gateway from Lambda/EC2
Now that we have our API ready to go it’s time to put it at a test. The code that we will demonstrate below is universal and is supported by:
- Lambda
- EC2 Instances
- Locally as long as the right permissions are available and there’s no firewall restrictions
To demonstrate this we will try to invoke an GET endpoint and see if it’s giving us back a 200 HTTP Response. If we get that back it would indicate that the API is functional and fully working and can be accessed by your external apps or web frameworks using the toolkit of your preference.
For simplicity we will be using a Python library called requests that includes some boiler plate code that lets you make HTTP requests to your backend server. Earlier we discussed how this is installed in our virtual environment.
import requests
import pprint
url = 'https://41a9v8ivkj.execute-api.us-east-1.amazonaws.com/dev/person'
ret = requests.get(url)
print ('Checking if status code is 200: ', ret.status_code)
print ('Response headers: ', ret.headers)
In the code above you may notice that we are not using the Authorization API Key that we previously created. This is to keep the code simple the authentication part has been removed from the API endpoint. If you’d like to know how to test this please drop me a line below and I’ll add some code for this.
main (venv) alex@DYNAMH ~/code/unbiased-coder/python-boto3-api-gateway-guide > python ./test_api_gateway.py
Checking if status code is 200: 200
Response headers: {'Date': 'Tue, 02 Nov 2021 13:45:37 GMT', 'Content-Type': 'application/json', 'Content-Length': '0', 'Connection': 'keep-alive', 'x-amzn-RequestId': '2795cbd8-efab-47b9-bb48-d073f320637c', 'x-amz-apigw-id': 'ILcYLHCNoAMFrtg='}
As it can be seen above our API endpoint works successfully and is returning back to us HTTP 200 which indicates a successful query.
Conclusion
If you found Python Boto3 API Gateway: Post, Get, Lambda/EC2, Models, Auth useful and you think it may have helped you please drop me a cheer below I would appreciate it.
If you have any questions, comments please post them below I check periodically and try to answer them in the priority they come in. Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.
What API solution do you use in your projects?
If you would like to learn more about AWS interfaces please take a look at the articles below: