Introduction
We will go over a Python AWS Boto3 Load Balancer Guide.
Did you know that load balancer changes can be done with Python?
We will break down this in the following sections:
- Why Is it Important To Control AWS Load Balancers Programmatically
- How to setup your environment to control AWS Load Balancer networks programmatically
- How to list all AWS Load Balancers
- How To Add and Delete AWS Load Balancers
I have used this successfully in various projects and it works very well and has saved me a ton of time getting going with managing my load balancers in a programmatic way. We will cover how to do this in Python using the Boto3 library framework.
We will go point by point on getting you up and running in less than 5mins, you do not need to have any programming knowledge to use the tool we are going to be developing here but know Python is preferable if you would like to modify the code for your needs.
This is a complete guide and should cover all your questions on managing load balancers in an AWS environment.
All code and examples on how to do this can be found in the Github link here.
When To Programmatically Control Your AWS Load Balancers
Before I dive into any code and implementation I want to give you a list of reasons of why this may or may not be useful for your needs. Basically whenever programming is involved you need to understand that the nature of your requirements but also capabilities increase so lets go over some reasons for this.
- You can setup conditional load balancer creations allowing Python to extend the logic
- You can integrate this in a system that performs devops operations such as a pipeline
- You can do bulk operations which are hard to do manually over the console such as deleting mass Load Balancers in your AWS account
- This can be part of a batch job that’s scheduled to run periodically
- You can use this code to automatically check certain things in your AWS account and set filters or rules on how your load balancers are created using the enumeration method we will describe below
- You can set your own limits on how many load balancers and what IP ranges you want to use and enforce them by periodically checking and even removing them automatically if they do not satisfy your requirements.
As you can see the list is rather big but there’s some cases where you may not want to use programmatic access to manage your load balancers. A list of those is below:
- You have a small amount of load balancers that you rarely change
- Your environment is highly critical and all changes should be done manually
- You do not need to perform bulk operations or add programming logical conditions to it
- You are just happy with the default load balancers that AWS provisions for you in your EC2 section
I’m sure I may have missed some on the lists above and if you do spot something please send me a note and I will be sure to update it.
How To Setup Boto3 To Manage Your Load Balancers
We will start by going over how to setup your system in order to run the Boto3 code which we will be outlining below. Since the Boto3 AWS library is a Python library we will be leveraging here a virtual environment to install our Python libraries to keep the main system clean from any dependencies.
For this I have made the process fairly easy for you by including the needed requirements file which contains all the packages you need to get going. Below you will find an outline of the commands you need to execute to get the environment setup and ready to go. You can also use conda if you prefer that but to keep things simple and lean I’m using directly virtualenv here.
$ virtualenv venv
created virtual environment CPython3.9.12.final.0-64 in 208ms
creator CPython3Posix(dest=/Users/alex/code/unbiased/python-boto3-load-balancer/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/alex/Library/Application Support/virtualenv)
added seed packages: pip==22.1.2, setuptools==62.6.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
$ pip install -r requirements.txt
Collecting boto3
....
Installing collected packages: urllib3, six, python-dotenv, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.24.37 botocore-1.27.37 jmespath-1.0.1 python-dateutil-2.8.2 python-dotenv-0.20.0 s3transfer-0.6.0 six-1.16.0 urllib3-1.26.11
$ python
>>> import dotenv
>>> dotenv.__all__
['get_cli_string', 'load_dotenv', 'dotenv_values', 'get_key', 'set_key', 'unset_key', 'find_dotenv', 'load_ipython_extension']
>>> import boto3
>>> boto3.__version__
'1.24.37'
As you can see from the code above we basically installed two packages:
- dotenv: This contains all the environment variables were will be using in our code such as the AWS access and secret key and also the region of AWS we will be using.
- boto3: This is the AWS library that contains the code we need to connect and interact with our load balancers in the AWS account.
To test the installation we simply run the version and some functions for dotenv to ensure it got installed properly, you can skip that test if you want it’s just a precaution here.
Now that we have the Python packages installed we need to go over some helper libraries we have implemented in order to wrap the Boto3 load balancers. Lets start by checking the code which is shown below and then we will do an analysis on the function helpers we wrote.
import os
from urllib import response
import boto3
import pprint
from dotenv import load_dotenv
def get_aws_keys():
load_dotenv()
return os.getenv('AWS_ACCESS_KEY'), os.getenv('AWS_SECRET_KEY')
def init_aws_session():
access_key, secret_key = get_aws_keys()
return boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name=os.getenv('AWS_REGION'))
def ec2_get_vpc_list():
session = init_aws_session()
ec2 = session.client('ec2')
response = ec2.describe_vpcs()
return response['Vpcs']
def elb_create_target_group(target_group_name, vpc_id):
session = init_aws_session()
elb = session.client('elbv2')
response = elb.create_target_group(Name=target_group_name, Protocol='HTTP', Port=80, VpcId=vpc_id)
return response
def ec2_get_subnet_list():
session = init_aws_session()
ec2 = session.client('ec2')
response = ec2.describe_subnets()
return response['Subnets']
def ec2_get_security_group_list():
session = init_aws_session()
ec2 = session.client('ec2')
response = ec2.describe_security_groups()
return response['SecurityGroups']
The first few functions that initialize the AWS session have been discussed extensively in a previous article that I wrote which you can find below:
- Boto3 Session: Setup Profile, Create, Close and Mock sessions
- Python AWS Boto3 VPC Guide
- Python AWS Boto3 Subnet Guide
- Python AWS Boto3 Security Group Guide
So for simplicity and to avoid repetition here I’m going to cover the new functions that are related to Load Balancers. The function names are made in such as a way to be self explanatory on what they do so I’m going to focus mainly on the functionality.
The first thing you would notice that’s common in all the functions is that we initialize an AWS session which we use later to perform the Load Balancer operations. This is shown in the init_aws_session function. Once the session is successfully initialized (this is where authentication happens too) then we need to acquire a client for the EC2 code base from the Boto3 library. The EC2 portion has a subset in it which contains the load balancer related wrappers which we will be using below.
elb_create_target_group: This function is a helper that lets us create a target group where our Load Balancer will be associated with. By doing this you can basically have one more instances such as EC2 or Lambdas tied to the load balancer. This function takes two parameters:
- VPC ID: The VPC Identifier we want to link this load balancer with
- Target Group Name: The name of the target group we will be creating
How To List Load Balancers With Boto3
In this section we will cover how to list the load balancers that exist in your AWS account using Boto3. The code below contains everything you need to perform this operation. This works in the following way:
- Initialize AWS session using the helper library we explained earlier in this article
- Once the session established get a handle for the ELBv2 part of the EC2 Boto3 library
- Then use the describe_load_balancers function to get a list of available load balancers
- Print out the result in this case we only need the Load Balancers part to be printed out.
The code to do this is shown below.
import pprint
from boto3_helper import init_aws_session
session = init_aws_session()
elb = session.client('elbv2')
response = elb.describe_load_balancers()
pprint.pprint(response['LoadBalancers'])
The next step is to execute the code in our AWS account and see what results it gives us.
$ python ./boto3-load-balancers-list.py
[{'AvailabilityZones': [{'LoadBalancerAddresses': [],
'SubnetId': 'subnet-03f55e65',
'ZoneName': 'us-east-1d'},
....
{'LoadBalancerAddresses': [],
'SubnetId': 'subnet-7ebb454f',
'ZoneName': 'us-east-1e'},
{'LoadBalancerAddresses': [],
'SubnetId': 'subnet-81f08bcc',
'ZoneName': 'us-east-1b'}],
'CanonicalHostedZoneId': 'Z35SXDOTRQ7X7K',
'CreatedTime': datetime.datetime(2022, 7, 25, 21, 32, 8, 30000, tzinfo=tzutc()),
'DNSName': 'internal-UnbiasedCoderLoadBalancer-998375431.us-east-1.elb.amazonaws.com',
'IpAddressType': 'ipv4',
'LoadBalancerArn': 'arn:aws:elasticloadbalancing:us-east-1:033533902081:loadbalancer/app/UnbiasedCoderLoadBalancer/be763399dd3bda54',
'LoadBalancerName': 'UnbiasedCoderLoadBalancer',
'Scheme': 'internal',
'SecurityGroups': ['sg-02dee6d59d2d17baa'],
'State': {'Code': 'active'},
'Type': 'application',
'VpcId': 'vpc-af9c5bd2'}]
As you can see above the describe function has returned a list of all the available load balancers with our account. There’s certain attributes here that are of importance that we need to outline below and you need to be aware of if you are going to work with load balancers.
- State: The state of your VPC is important to know because sometimes it may be disabled or turned off and this gives you visibility to it.
- Owner: The Identifier of the owner is also good to know for accounting reasons and if you are trying to debug something.
- Load Balancer ARN: This is a unique identifier that is used across the AWS infrastructure to identify our load balancer.
- DNS Name: In order to connect to the load balancer you need to use this DNS address
- Availability Zones: This is often overlooked but it specifies which zones the load balancer is available on, typically you will want to use more than one here.
The rest of the options while important I do not believe play a major role for your basic management of your load balancers using Boto3.
How To Create A Load Balancer With Boto3
Moving on we are going to cover how to add a load balancer using Boto3. In order to do this we will be breaking down the code logic of how this works and what steps we need to take in order to successfully create a load balancer using Boto3.
- We will first need to get a security group list and find which of those has a standard listener info with the following attributes:
- HTTP Port 80 and/or 443
- Protocol: TCP
- Once we identify the security group by iterating and searching via all of them we need to save the security group identifier to use later on in the code
- The next step is to find the VPC Identifier that’s associated with that subnet and network and for this we will be leveraging the functions we had in our wrappers previously
- So now we have an association between the three following important parts:
- VPC ID
- Subnet ID
- Security Group ID
- Using the information above we proceed with creating a sample target group which will then be tied with our load balancer
- Subsequently the final call that does all the magic and ties the attributes together between our subnet group, target group is the create_load_balancer function, this function returns a load balancer ARN which is also the unique identifier that we can use later on to verify it’s creation.
import pprint
from boto3_helper import *
# find a security group that allows port 80 and tcp
security_groups = ec2_get_security_group_list()
for security_group in security_groups:
security_group_ip_perms = security_group['IpPermissions']
for security_group_ip_perm in security_group_ip_perms:
if security_group_ip_perm['IpProtocol'] == 'tcp' and security_group_ip_perm['FromPort'] == 80:
vpc_id = security_group['VpcId']
security_group_id = security_group['GroupId']
break
# find subnet and VPC ID associated with security group
subnet_list = ec2_get_subnet_list()
subnet_id_list = []
for subnet in subnet_list:
if subnet['VpcId'] == vpc_id:
subnet_id_list.append(subnet['SubnetId'])
print (subnet_id_list, vpc_id, security_group_id)
# Create a target group
target_group = elb_create_target_group('unbiased-coder-target-group', vpc_id)
target_group_arn = target_group['TargetGroups'][0]['TargetGroupArn']
session = init_aws_session()
elb = session.client('elbv2')
response = elb.create_load_balancer(
Name='UnbiasedCoderLoadBalancer',
Subnets = subnet_id_list,
SecurityGroups=[
security_group_id,
],
Scheme='internal',
Type='application',
IpAddressType='ipv4',
)
pprint.pprint(response)
In the code above you can see we are basically adding the load balancer for port 80/HTTP on one of the private subnets that exists in our account.
To further demonstrate how this works and see the response that AWS sends back to us when it gets added we will be executing the code to create this new Load Balancer with the Boto3 code wrapper we implemented.
$ python ./boto3-load-balancer-add.py
{'LoadBalancers': [{'AvailabilityZones': [{'LoadBalancerAddresses': [],
'SubnetId': 'subnet-17f64736',
'ZoneName': 'us-east-1a'},
...
{'LoadBalancerAddresses': [],
'SubnetId': 'subnet-81f08bcc',
'ZoneName': 'us-east-1b'}],
'CanonicalHostedZoneId': 'Z35SXDOTRQ7X7K',
'CreatedTime': datetime.datetime(2022, 7, 27, 19, 19, 17, 110000, tzinfo=tzutc()),
'DNSName': 'internal-UnbiasedCoderLoadBalancerNew-127481148.us-east-1.elb.amazonaws.com',
'IpAddressType': 'ipv4',
'LoadBalancerArn': 'arn:aws:elasticloadbalancing:us-east-1:033533902081:loadbalancer/app/UnbiasedCoderLoadBalancerNew/229e0cd4748d1d6a',
'LoadBalancerName': 'UnbiasedCoderLoadBalancerNew',
'Scheme': 'internal',
'SecurityGroups': ['sg-02dee6d59d2d17baa'],
'State': {'Code': 'provisioning'},
'Type': 'application',
'VpcId': 'vpc-af9c5bd2'}]}
$ python ./boto3-load-balancers-list.py|grep "arn:aws:elasticloadbalancing:us-east-1:033533902081:loadbalancer/app/UnbiasedCoderLoadBalancerNew/229e0cd4748d1d6a"
'LoadBalancerArn': 'arn:aws:elasticloadbalancing:us-east-1:033533902081:loadbalancer/app/UnbiasedCoderLoadBalancerNew/229e0cd4748d1d6a',
As you can see above AWS successfully added the load balancer to our list and returned to us the unique load balancer arn allocated to it, in this case: arn:aws:elasticloadbalancing:us-east-1:033533902081:loadbalancer/app/UnbiasedCoderLoadBalancerNew/229e0cd4748d1d6a”. As an extra verification we will be leveraging the code to list all load balancers in our system and basically grep specifically for this newly created load balancer arn, to see if it exists in our system. Above it’s visible that the load balancer exists and it was successfully added to it.
How To Delete Load Balancer Using Boto3
In this section we will demonstrate how to delete a load balancer network using Boto3. Since we already added a test load balancer earlier we will basically proceed into deleting it now and checking to see if it still exists in the list.
Similarly to the two earlier examples we will use the Boto3 delete load balancer wrapper code we wrote and we will be passing to it the load balancer ARN that we want to mark for deletion.
import pprint
from boto3_helper import init_aws_session
session = init_aws_session()
elb = session.client('elbv2')
response = elb.delete_load_balancer(LoadBalancerArn='arn:aws:elasticloadbalancing:us-east-1:033533902081:loadbalancer/app/UnbiasedCoderLoadBalancerNew/229e0cd4748d1d6a')
pprint.pprint(response)
The code above works similarly to what we did earlier but uses the delete_load_balancer function. That function takes as a parameter the load balancer ARN we found earlier which is going to use for deletion from the AWS account. The next step is to go ahead and run the load balancer deletion code and see what it results in.
$ python ./boto3-load-balancer-delete.py
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '262',
'content-type': 'text/xml',
'date': 'Wed, 27 Jul 2022 19:24:04 GMT',
'x-amzn-requestid': '08ab6cb4-12a3-4580-8213-6cdd9720e6e6'},
'HTTPStatusCode': 200,
'RequestId': '08ab6cb4-12a3-4580-8213-6cdd9720e6e6',
'RetryAttempts': 0}}
$ python ./boto3-load-balancers-list.py|grep "arn:aws:elasticloadbalancing:us-east-1:033533902081:loadbalancer/app/UnbiasedCoderLoadBalancerNew/229e0cd4748d1d6a"
[empty]
As you can see above the command succeeded with a 200 HTTP return code meaning the record of the load balancer we provided was deleted from the AWS account. Since we are adding extra verification we will again check our list for the load balancers to see if it exists in the list. Running the command shows that the load balancer has disappeared and is no longer there indicating.
The way we verified this is by grep’ing the output of the list load balancers command for the ARN: arn:aws:elasticloadbalancing:us-east-1:033533902081:loadbalancer/app/UnbiasedCoderLoadBalancerNew/229e0cd4748d1d6a which executed and did not return any results. With this we were able to successfully demonstrate how to remove a load balancer using Boto3 from the AWS account.
Conclusion
We were able to successfully go over a Python AWS Boto3 Load Balancer Guide, hopefully I answered any questions you may have had and helped you get started on your quest on managing your load balancers programmatically in your AWS account.
If you found this useful and you think it may have helped you please drop me a cheer below I would appreciate it.
If you have any questions, comments please post them below or send me a note on my twitter. I check periodically and try to answer them in the priority they come in. Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.
Do you prefer to use the AWS console or Boto3 to manage your load balancers?
I personally still think both have their place and complement each other when working from the terminal or wanting to do customized stuff that require programming and needing to add logic to it.
If you would like to visit the official Python Boto3 documentation here.
If you would like to find more articles related to AWS services: