Introduction
We will go over a Python AWS Boto3 Security Groups Guide.
Did you know that Boto3 allows you to programmatically change the security groups?
We will break down this in the following sections:
- Why Is it Important To Control AWS Security Groups Programmatically
- How to setup your environment to control AWS Security Groups programmatically
- How to list/add and delete Security Groups
- How to setup ingress/egress security group rules using boto3
I have used this successfully in various projects and it works very well and has saved me a ton of time getting going with managing my security groups along with the egress/ingress rules in a programmatic way. We will cover how to do this in Python using the Boto3 library framework.
We will go point by point on getting you up and running in less than 5mins, you do not need to have any programming knowledge to use the tool we are going to be developing here but know Python is preferable if you would like to modify the code for your needs.
This is a complete guide and should cover all your questions on managing Security groups and ingress/egress rules with boto3 in an AWS environment.
All code and examples on how to do this can be found in the Github link here.
Why It Is Useful To Automate Control Of Your AWS Security Groups
Before I dive into any code and implementation I want to give you a list of reasons of why this may or may not be useful for your needs. Basically whenever programming is involved you need to understand that the nature of your requirements but also capabilities increase so lets go over some reasons for this.
- You can setup conditional Security group creations allowing Python to extend the logic
- You can integrate this in a system that performs devops operations such as a pipeline
- You can do bulk operations which are hard to do manually over the console such as deleting/adding a huge list of security groups in your AWS account
- This can be part of a batch job that’s scheduled to run periodically
- You can use this code to automatically check certain things in your AWS account and set filters or rules on how your VPCs are created using the enumeration method we will describe below
- You can set your own limits on how many security groups and what requirements your ingress/egress rules will have. This will also allow you to enforce company wide policies on the port numbers, network protocols and services that are allowed within your AWS account.
As you can see the list is rather big but there’s some cases where you may not want to use programmatic access to manage your security groups. A list of those is below:
- You have a small set of services in your AWS account and that rarely changes
- Your environment is highly critical and needs to be always available so all changes should be done manually
- You do not need to perform bulk operations or add programming logical conditions to it
- You are just happy with the default security groups that AWS provides to your accounts and services
I’m sure I may have missed some on the lists above and if you do spot something please send me a note and I will be sure to update it.
How To Setup Boto3 To Manage Security Groups And Rules
We will start by going over how to setup your system in order to run the Boto3 code which we will be outlining below. Since the Boto3 AWS library is a Python library we will be leveraging here a virtual environment to install our Python libraries to keep the main system clean from any dependencies.
For this I have made the process fairly easy for you by including the needed requirements file which contains all the packages you need to get going. Below you will find an outline of the commands you need to execute to get the environment setup and ready to go. You can also use conda if you prefer that but to keep things simple and lean I’m using directly virtualenv here.
$ virtualenv venv
created virtual environment CPython3.9.12.final.0-64 in 208ms
creator CPython3Posix(dest=/Users/alex/code/unbiased/python-boto3-load-balancer/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/alex/Library/Application Support/virtualenv)
added seed packages: pip==22.1.2, setuptools==62.6.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
$ pip install -r requirements.txt
Collecting boto3
....
Installing collected packages: urllib3, six, python-dotenv, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.24.37 botocore-1.27.37 jmespath-1.0.1 python-dateutil-2.8.2 python-dotenv-0.20.0 s3transfer-0.6.0 six-1.16.0 urllib3-1.26.11
$ python
>>> import dotenv
>>> dotenv.__all__
['get_cli_string', 'load_dotenv', 'dotenv_values', 'get_key', 'set_key', 'unset_key', 'find_dotenv', 'load_ipython_extension']
>>> import boto3
>>> boto3.__version__
'1.24.37'
As you can see from the code above we basically installed two packages:
- dotenv: This contains all the environment variables were will be using in our code such as the AWS access and secret key and also the region of AWS we will be using.
- boto3: This is the AWS library that contains the code we need to connect and interact with our Security groups and security rules in the AWS account.
To test the installation we simply run the version and some functions for dotenv to ensure it got installed properly, you can skip that test if you want it’s just a precaution here.
Now that we have the Python packages installed we need to go over some helper libraries we have implemented in order to wrap the Boto3 security group and rules. Lets start by checking the code which is shown below and then we will do an analysis on the function helpers we implemented.
import os
from urllib import response
import boto3
import pprint
from dotenv import load_dotenv
def get_aws_keys():
load_dotenv()
return os.getenv('AWS_ACCESS_KEY'), os.getenv('AWS_SECRET_KEY')
def init_aws_session():
access_key, secret_key = get_aws_keys()
return boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name=os.getenv('AWS_REGION'))
def ec2_get_security_group_list():
session = init_aws_session()
ec2 = session.client('ec2')
response = ec2.describe_security_groups()
return response['SecurityGroups']
def ec2_add_security_group(desc, grp_name, vpc_id):
session = init_aws_session()
ec2 = session.client('ec2')
response = ec2.create_security_group(Description=desc, GroupName=grp_name, VpcId=vpc_id)
return response
def ec2_delete_security_group(grp_id):
session = init_aws_session()
ec2 = session.client('ec2')
response = ec2.delete_security_group(GroupId=grp_id)
return response
def ec2_add_security_group_rule(grp_id, proto, start_port, end_port, ip_range):
session = init_aws_session()
ec2 = session.client('ec2')
response = ec2.authorize_security_group_egress(
GroupId=grp_id,
IpPermissions=[{
'IpProtocol': proto,
'FromPort': start_port,
'ToPort': end_port,
'IpRanges': [{
'CidrIp': ip_range
}]
}])
return response
def ec2_delete_security_group_rule(security_grp_id, security_grp_rule_id):
session = init_aws_session()
ec2 = session.client('ec2')
response = ec2.revoke_security_group_egress(GroupId=security_grp_id, SecurityGroupRuleIds=[security_grp_rule_id])
return response
The first few functions that initialize the AWS session have been discussed extensively in a previous article that I wrote which you can find below:
So for simplicity and to avoid repetition here I’m going to cover the new functions that are related to security group and rule management. The function names are made in such as a way to be self explanatory on what they do so I’m going to focus mainly on the functionality.
The first thing you would notice that’s common in all the functions is that we initialize an AWS session which we use later to perform the security group operations. This is shown in the init_aws_session function. Once the session is successfully initialized (this is where authentication happens too) then we need to acquire a client for the EC2 code base from the Boto3 library. The EC2 portion has a subset in it which contains the security group and rules wrappers which we will be using below.
ec2_add_security_group: The first function we will be talking about is adding a security group in AWS using Boto3. This function basically takes three arguments:
- VPC ID: This is the identifier of the VPC network we want to associate the security group with
- Description: This is simply a description to identify your group easier
- Group name: This is the name of the group we will be creating (the new one)
Using the parameters above we can add a security group using boto3.
ec2_delete_security_group: Similarly to adding we also implement the security group deletion function. This function basically does the opposite and just removes a security group from our list. Each security group has a unique identifier associated with it which is called security group ID. Using the security group ID we can easily manage a particular security group including deleting it. When a security group gets first added you can see what unique identifier AWS associated for it and save it, or you can use the method which we will talk about next.
ec2_get_security_group_list: The Get security list lets you list all AWS security groups in your AWS account using Boto3. Basically this is useful if you are trying to find particular details about a security group such as finding the following information:
- Egress rules
- Ingress rules
- Security group information such as the ID and the group name
ec2_add_security_group_rule: Further to security groups we need to be able to add new security rules associated with it. In this case we will be demonstrating a function that adds a security group rule this can be of two types ingress and egress. Both rule types work exactly the same way the difference is that one is for inbound traffic (ingress) and egress (outbound traffic). The arguments we need to pass to this helper are the following:
- Group ID: This is the security group identifier we will be associating the security rule with
- Protocol: The network protocol that we are specifying this, for example this can be tcp or any if you don’t want to specify one protocol alone
- Start Port: The source or destination network port if there’s one to specify in the range that the range starts with
- End Port: The destination or source network port if there’s one to specify or a range the the range ends with
- IP Range: Finally the IP address range if there’s one to specify, like the everything else this can be 0.0.0.0/0 if you want to allow everything
ec2_delete_security_group_rule: Finally the last function we will be implementing here as a helper is the deletion of security group rules. Again this works for both the ingress and egress traffic. More particularly we can delete the rules we previously created. For this to work we need specify two parameters simply the security group identifier and the security rule identifier we want to delete.
How To List Security Groups With Boto3
Now that we covered the basic Boto3 Security Group function wrappers lets start implementing some examples and interacting with our AWS account. For this you may see specific IDs associated in the code but that’s just for simplicity and only work with my AWS account. In order to run this in your system you will need to keep track of what exists and save it or use the security group list Boto3 code below to enumerate everything and pick one to manipulate.
import pprint from boto3_helper import ec2_get_security_group_list security_groups = ec2_get_security_group_list() pprint.pprint(security_groups)
The code above basically uses the library we implemented earlier that lists all security groups in your AWS account using Boto3. We simply run the function and print out the results of our security groups. An example execution of this is in my AWS account and can be seen below.
$ python ./boto3-security-groups-list.py
[{'Description': 'launch-wizard-3 created 2022-04-27T12:58:43.104Z',
'GroupId': 'sg-0a4791007be830b56',
'GroupName': 'launch-wizard-3',
'IpPermissions': [{'FromPort': 80,
'IpProtocol': 'tcp',
'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
'Ipv6Ranges': [],
'PrefixListIds': [],
'ToPort': 80,
'UserIdGroupPairs': []},
{'FromPort': 22,
'IpProtocol': 'tcp',
'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
'Ipv6Ranges': [],
'PrefixListIds': [],
'ToPort': 22,
'UserIdGroupPairs': []},
{'FromPort': 443,
'IpProtocol': 'tcp',
'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
'Ipv6Ranges': [],
'PrefixListIds': [],
'ToPort': 443,
'UserIdGroupPairs': []}],
'IpPermissionsEgress': [{'FromPort': 9000,
'IpProtocol': 'tcp',
'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
'Ipv6Ranges': [],
'PrefixListIds': [],
'ToPort': 9001,
'UserIdGroupPairs': []},
{'IpProtocol': '-1',
'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
'Ipv6Ranges': [],
'PrefixListIds': [],
'UserIdGroupPairs': []}],
'OwnerId': '033533902081',
'VpcId': 'vpc-af9c5bd2'},
....
]
To cut the output I just showed the first few networks that exist in my account. As you can see the security groups have a lot more information associated with them besides the rules and a unique identifier. Since this is beyond the scope of this article I’m not going to go over every single one but if you want me I can do it in the comments below so drop me a line. However I do want to explain a few fields here that may be useful for you.
- VPC ID: The VPC network ID where your your security groups are associated with.
- Owner: The Identifier of the owner is also good to know for accounting reasons and if you are trying to debug something.
- Rules: The in and outbound security group rules for your network traffic
The rest of the options while important I do not believe play a major role for your basic management of your security groups using Boto3.
How To Create A Security Group With Boto3
Moving on we are going to cover how to add a security group network using Boto3. Similar to before the logic to this is basically to invoke the function we implemented on our Boto3 security group function wrapper. The only difference here is that when invoking our function to add a security groups to our AWS account we are also going to need to pass in the arguments we discussed above such as the group name, the description and the VPC ID we want to associate it with.
import pprint
from boto3_helper import ec2_add_security_group
result = ec2_add_security_group('Unbiased Coder Security_Group', 'UnbiasedCoderSecurityGrp', 'vpc-af9c5bd2')
pprint.pprint(result)
In the code above you can see we are basically adding the security group name UnbiasedCoderSecurityGrp associating with the VPC ID: vpc-af9c5bd2.
To further demonstrate how this works and see the response that AWS sends back to us when it gets added we will be executing the code to create this new security group with the Boto3 code wrapper we implemented.
$ python ./boto3-security-groups-add.py
{'GroupId': 'sg-0148ad8f0314a586b',
'ResponseMetadata': {'HTTPHeaders': {'cache-control': 'no-cache, no-store',
'content-length': '283',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Wed, 27 Jul 2022 14:07:59 GMT',
'server': 'AmazonEC2',
'strict-transport-security': 'max-age=31536000; '
'includeSubDomains',
'x-amzn-requestid': '081156e0-93f1-46c0-ac24-c659158de7bf'},
'HTTPStatusCode': 200,
'RequestId': '081156e0-93f1-46c0-ac24-c659158de7bf',
'RetryAttempts': 0}}
$ python ./boto3-security-groups-list.py|grep Description
[{'Description': 'launch-wizard-3 created 2022-04-27T12:58:43.104Z',
{'Description': 'Unbiased Coder Security Group',
{'Description': 'launch-wizard-1 created 2022-02-16T18:30:13.588+02:00',
{'Description': 'default VPC security group',
{'Description': 'launch-wizard-2 created 2022-02-16T18:38:44.708+02:00',
{'Description': 'Unbiased Coder Security_Group',
{'Description': 'default VPC security group',
{'Description': 'launch-wizard-4 created 2022-05-10T17:12:19.044Z',
As you can see above AWS successfully added the security group to our list and returned to us the unique security group identifier to allocated to it, in this case: sg-0148ad8f0314a586b. As an extra verification we will be leveraging the code to list all security groups in our system and basically grep specifically for this new security group description to see if it exists in our system. Above it’s visible that the unbiased coder security group description we specified was successfully added.
How To Delete Security Group With Boto3
In this section we will demonstrate how to delete a Security Group using Boto3. Since we already added a test security group earlier we will basically proceed into deleting it now and checking to see if it still exists in the list.
Similarly to the two earlier examples we will use the Boto3 delete security group wrapper code we wrote and we will be passing to it the security group ID that we want to mark for deletion.
import pprint
from boto3_helper import ec2_delete_security_group
result = ec2_delete_security_group('sg-0148ad8f0314a586b')
pprint.pprint(result)
We basically send the command to the AWS service to delete the security group and then print out the result. The execution of this is shown below.
$ python ./boto3-security-groups-delete.py
{'ResponseMetadata': {'HTTPHeaders': {'cache-control': 'no-cache, no-store',
'content-length': '239',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Wed, 27 Jul 2022 14:09:50 GMT',
'server': 'AmazonEC2',
'strict-transport-security': 'max-age=31536000; '
'includeSubDomains',
'x-amzn-requestid': '87ecb527-b02b-47e5-95f7-bd279199155c'},
'HTTPStatusCode': 200,
'RequestId': '87ecb527-b02b-47e5-95f7-bd279199155c',
'RetryAttempts': 0}}
$ python ./boto3-security-groups-list.py|egrep "Description|GroupId"
[{'Description': 'launch-wizard-3 created 2022-04-27T12:58:43.104Z',
'GroupId': 'sg-0a4791007be830b56',
{'Description': 'Unbiased Coder Security Group',
'GroupId': 'sg-0d7465e50c9b66f48',
{'Description': 'launch-wizard-1 created 2022-02-16T18:30:13.588+02:00',
'GroupId': 'sg-0895d32733616dc83',
{'Description': 'default VPC security group',
'GroupId': 'sg-03f2b6fc19e859296',
'UserIdGroupPairs': [{'GroupId': 'sg-03f2b6fc19e859296',
{'Description': 'launch-wizard-2 created 2022-02-16T18:38:44.708+02:00',
'GroupId': 'sg-02dee6d59d2d17baa',
{'Description': 'default VPC security group',
'GroupId': 'sg-eb12a8d5',
'UserIdGroupPairs': [{'GroupId': 'sg-eb12a8d5',
{'Description': 'launch-wizard-4 created 2022-05-10T17:12:19.044Z',
'GroupId': 'sg-04e81bd8b5e3c5d5c',
As you can see above the command succeeded with a 200 HTTP return code meaning the record of the security group we provided was deleted from the AWS account. Since we are adding extra verification we will again check our list for the security group ID and descriptions, to see if it exists in the list. Running the command shows that the security group has disappeared and is no longer there indicating we were successfully able to demonstrate how to remove a security group network using Boto3 from the AWS account.
How To Create A Security Group Ingress/Egress Rule With Boto3
Moving on we are going to cover how to add a security group ingress/egress using Boto3. Similar to before the logic to this is basically to invoke the function we implemented on our Boto3 security group rule for egress/ingress function wrapper. The only difference here is that when invoking our function to add a security group rule to our AWS account we are also going to need to pass in the arguments we discussed above such as:
- Security group ID where the egress/ingress rule needs to be added on, in this case: sg-0a4791007be830b56
- Network protocol in this case TCP
- Port start and end range for the rule: 9000-9001
- IP network range in this case we allow everything so we use 0.0.0.0/0
import pprint
from boto3_helper import ec2_add_security_group_rule
result = ec2_add_security_group_rule('sg-0a4791007be830b56', 'tcp', 9000, 9001, '0.0.0.0/0')
pprint.pprint(result)
To further demonstrate how this works and see the response that AWS sends back to us when it gets added we will be executing the code to create this new security group rule for either ingress or egress using the Boto3 code wrapper we implemented.
$ python ./boto3-security-group-rule-add.py
{'ResponseMetadata': {'HTTPHeaders': {'cache-control': 'no-cache, no-store',
'content-length': '720',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Wed, 27 Jul 2022 14:16:29 GMT',
'server': 'AmazonEC2',
'strict-transport-security': 'max-age=31536000; '
'includeSubDomains',
'x-amzn-requestid': 'e3fdfc2a-af0e-4c8e-9ac7-14f2a99afb26'},
'HTTPStatusCode': 200,
'RequestId': 'e3fdfc2a-af0e-4c8e-9ac7-14f2a99afb26',
'RetryAttempts': 0},
'Return': True,
'SecurityGroupRules': [{'CidrIpv4': '0.0.0.0/0',
'FromPort': 9000,
'GroupId': 'sg-0a4791007be830b56',
'GroupOwnerId': '033533902081',
'IpProtocol': 'tcp',
'IsEgress': True,
'SecurityGroupRuleId': 'sgr-02f2ba16855db024f',
'ToPort': 9001}]}
$ python ./boto3-security-groups-list.py
[{'Description': 'launch-wizard-3 created 2022-04-27T12:58:43.104Z',
'GroupId': 'sg-0a4791007be830b56',
'GroupName': 'launch-wizard-3',
'IpPermissions': [
....
'IpPermissionsEgress': [{'FromPort': 9000,
'IpProtocol': 'tcp',
'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
'Ipv6Ranges': [],
'PrefixListIds': [],
'ToPort': 9001,
'UserIdGroupPairs': []},
{'IpProtocol': '-1',
'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
'Ipv6Ranges': [],
'PrefixListIds': [],
'UserIdGroupPairs': []}],
As you can see above AWS successfully added the security group rule to the security group to our list and returned to us and returned a unique security group rule identifier to allocated to it. As an extra verification we will be leveraging the code to list all security groups in our system to see if the new rule was added in our system. As shown above we can see the 9000-9001 port range we added indicating this.
How To Delete Security Group Ingress/Egress Rule With Boto3
In this section we will demonstrate how to delete a Security Group ingress/egress rule using Boto3. Since we already added a test security group rule earlier we will basically proceed into deleting it now and checking to see if it still exists in the list.
Similarly to the two earlier examples we will use the Boto3 delete security group rule wrapper code we wrote and we will be passing to it the security group rule ID that we want to mark for deletion.
import pprint
from boto3_helper import ec2_delete_security_group_rule
result = ec2_delete_security_group_rule('sg-0a4791007be830b56', 'sgr-02f2ba16855db024f')
pprint.pprint(result)
We basically send the command to the AWS service to delete the security group rule and then print out the result. The execution of this is shown below.
$ python ./boto3-security-group-rule-delete.py
{'ResponseMetadata': {'HTTPHeaders': {'cache-control': 'no-cache, no-store',
'content-length': '251',
'content-type': 'text/xml;charset=UTF-8',
'date': 'Wed, 27 Jul 2022 14:24:55 GMT',
'server': 'AmazonEC2',
'strict-transport-security': 'max-age=31536000; '
'includeSubDomains',
'x-amzn-requestid': 'eebcf665-cda7-489e-b926-fea30d545a7e'},
'HTTPStatusCode': 200,
'RequestId': 'eebcf665-cda7-489e-b926-fea30d545a7e',
'RetryAttempts': 0},
'Return': True}
$ python ./boto3-security-groups-list.py|grep 9000
[empty]
As you can see above the command succeeded with a 200 HTTP return code meaning the record of the security group ingress/egress rule we provided was deleted from the AWS account. We do this by checking specifically for the rule port number in this case 9000 which returns no results meaning the ingress/egress rule got deleted. We were successfully able to demonstrate how to remove a security group ingress/egress rule using Boto3 from the AWS account.
Conclusion
We were able to successfully go over a Python AWS Boto3 Security Groups Guide, hopefully I answered any questions you may have had and helped you get started on your quest on managing your security groups and rules programmatically in your AWS account.
If you found this useful and you think it may have helped you please drop me a cheer below I would appreciate it.
If you have any questions, comments please post them below or send me a note on my twitter. I check periodically and try to answer them in the priority they come in. Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.
How do you change your security rules programmatically or via AWS console?
I personally still think both have their place and complement each other when working from the terminal or wanting to do customized stuff that require programming and needing to add logic to it.
If you would like to visit the official Python Boto3 documentation here.
If you would like to find more articles related to AWS services: