Python AWS Boto3 VPC Guide

AWS, EC2, S3, VPC/Subnet

Introduction

Python AWS Boto3 VPC Guide
Python AWS Boto3 VPC Guide

We will go over Python AWS Boto3 VPC Guide.

Did you know that Boto3 offers a fully managed way to control your VPCs programmatically?

We will break down this in the following sections:

  • Why Is it Important To Control AWS VPCs Programmatically
  • How to setup your environment to control AWS VPC networks programmatically
  • How to list all AWS VPCs
  • How To Add and Delete AWS VPCs

I have used this successfully in various projects and it works very well and has saved me a ton of time getting going with managing my VPCs in a programmatic way. We will cover how to do this in Python using the Boto3 library framework.

We will go point by point on getting you up and running in less than 5mins, you do not need to have any programming knowledge to use the tool we are going to be developing here but know Python is preferable if you would like to modify the code for your needs.

This is a complete guide and should cover all your questions on managing VPCs in an AWS environment.

All code and examples on how to do this can be found in the Github link here.

When To Programmatically Control Your AWS VPC Networks

Before I dive into any code and implementation I want to give you a list of reasons of why this may or may not be useful for your needs. Basically whenever programming is involved you need to understand that the nature of your requirements but also capabilities increase so lets go over some reasons for this.

  • You can setup conditional VPC creations allowing Python to extend the logic
  • You can integrate this in a system that performs devops operations such as a pipeline
  • You can do bulk operations which are hard to do manually over the console such as deleting mass VPCs in your AWS account
  • This can be part of a batch job that’s scheduled to run periodically
  • You can use this code to automatically check certain things in your AWS account and set filters or rules on how your VPCs are created using the enumeration method we will describe below
  • You can set your own limits on how many VPCs and what IP ranges you want to use and enforce them by periodically checking and even removing them automatically if they do not satisfy your requirements.

As you can see the list is rather big but there’s some cases where you may not want to use programmatic access to manage your VPC networks. A list of those is below:

  • You have a small amount of VPC networks that you rarely change
  • Your environment is highly critical and all changes should be done manually
  • You do not need to perform bulk operations or add programming logical conditions to it
  • You are just happy with the default VPC networks that AWS provides to you

I’m sure I may have missed some on the lists above and if you do spot something please send me a note and I will be sure to update it.

How To Setup Boto3 To Manage VPC

We will start by going over how to setup your system in order to run the Boto3 code which we will be outlining below. Since the Boto3 AWS library is a Python library we will be leveraging here a virtual environment to install our Python libraries to keep the main system clean from any dependencies.

For this I have made the process fairly easy for you by including the needed requirements file which contains all the packages you need to get going. Below you will find an outline of the commands you need to execute to get the environment setup and ready to go. You can also use condo if you prefer that but to keep things simple and lean I’m using directly virtualenv here.

$ virtualenv venv
created virtual environment CPython3.9.12.final.0-64 in 208ms
  creator CPython3Posix(dest=/Users/alex/code/unbiased/python-boto3-load-balancer/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/alex/Library/Application Support/virtualenv)
    added seed packages: pip==22.1.2, setuptools==62.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
$ pip install -r requirements.txt
Collecting boto3
....
Installing collected packages: urllib3, six, python-dotenv, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.24.37 botocore-1.27.37 jmespath-1.0.1 python-dateutil-2.8.2 python-dotenv-0.20.0 s3transfer-0.6.0 six-1.16.0 urllib3-1.26.11
$ python
>>> import dotenv
>>> dotenv.__all__
['get_cli_string', 'load_dotenv', 'dotenv_values', 'get_key', 'set_key', 'unset_key', 'find_dotenv', 'load_ipython_extension']
>>> import boto3
>>> boto3.__version__
'1.24.37'

As you can see from the code above we basically installed two packages:

  • dotenv: This contains all the environment variables were will be using in our code such as the AWS access and secret key and also the region of AWS we will be using.
  • boto3: This is the AWS library that contains the code we need to connect and interact with our VPC networks in the AWS account.

To test the installation we simply run the version and some functions for dotenv to ensure it got installed properly, you can skip that test if you want it’s just a precaution here.

Now that we have the Python packages installed we need to go over some helper libraries we have implemented in order to wrap the Boto3 VPC connectivity. Lets start by checking the code which is shown below and then we will do an analysis on the function helpers we wrote.

import os
import boto3
import pprint
from dotenv import load_dotenv

def get_aws_keys():
    load_dotenv()
    return os.getenv('AWS_ACCESS_KEY'), os.getenv('AWS_SECRET_KEY')

def init_aws_session():
    access_key, secret_key = get_aws_keys()
    return boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name=os.getenv('AWS_REGION'))

def ec2_get_vpc_list():
    session = init_aws_session()
    ec2 = session.client('ec2')
    response = ec2.describe_vpcs()
    return response['Vpcs']

def ec2_add_vpc(ip_block):
    session = init_aws_session()
    ec2 = session.client('ec2')
    response = ec2.create_vpc(CidrBlock = ip_block, InstanceTenancy='default')
    return response['Vpc']

def ec2_delete_vpc(vpc_id):
    session = init_aws_session()
    ec2 = session.client('ec2')
    response = ec2.delete_vpc(VpcId=vpc_id)
    return response

The first few functions that initialize the AWS session have been discussed extensively in a previous article that I wrote which you can find below:

So for simplicity and to avoid repetition here I’m going to cover the new functions that are related to VPC. The function names are made in such as a way to be self explanatory on what they do so I’m going to focus mainly on the functionality.

The first thing you would notice that’s common in all the functions is that we initialize an AWS session which we use later to perform the VPC operations. This is shown in the init_aws_session function. Once the session is successfully initialized (this is where authentication happens too) then we need to acquire a client for the EC2 code base from the Boto3 library. The EC2 portion has a subset in it which contains the VPC related wrappers which we will be using below.

ec2_add_vpc: The first function we will be talking about is adding a VPC network in AWS using Boto3. This function basically takes as an argument the very much needed IP address block we will be using to allocate to it. This can be a local network within AWS along with the net mask it has.

ec2_delete_vpc: Similarly to adding we also implement the VPC deletion function. This function basically does the opposite and just removes a VPC network from our list. Each VPC network has a unique identifier associated with it which is called VPC ID. Using the VPC ID we can easily manage a particular VPC including deleting it. When a VPC gets first added you can see what unique identifier AWS associated for it and save it, or you can use the method which we will talk about next.

ec2_get_vpc_list: The Get VPC list lets you list all AWS VPC networks in your AWS account using Boto3. Basically this is useful if you are trying to find particular details about a VPC such as the IP address block or a VPC ID to use later for deletion or modification.

How To List VPCs With Boto3

Now that we covered the basic Boto3 VPC function wrappers lets start implementing some examples and interacting with our AWS account. For this you may see specific IDs associated in the code but that’s just for simplicity and only work with my AWS account. In order to run this in your system you will need to keep track of what exists and save it or use the VPC list Boto3 code below to enumerate everything and pick one to manipulate.

import pprint
from boto3_helper import ec2_get_vpc_list

result = ec2_get_vpc_list()
pprint.pprint(result)

The code above basically uses the library we implemented earlier that lists all VPC networks in your AWS account using Boto3. We simply run the function and print out the results of our VPC networks. An example execution of this is in my AWS account and can be seen below.

$ python ./boto3-vpc-list.py
[{'CidrBlock': '172.31.0.0/16',
  'CidrBlockAssociationSet': [{'AssociationId': 'vpc-cidr-assoc-45b7672a',
                               'CidrBlock': '172.31.0.0/16',
                               'CidrBlockState': {'State': 'associated'}}],
  'DhcpOptionsId': 'dopt-831979f9',
  'InstanceTenancy': 'default',
  'IsDefault': True,
  'OwnerId': '033533902081',
  'State': 'available',
  'VpcId': 'vpc-af9c5bd2'},
 {'CidrBlock': '172.31.0.0/16',
  'CidrBlockAssociationSet': [{'AssociationId': 'vpc-cidr-assoc-0db97617b1fc326d1',
                               'CidrBlock': '172.31.0.0/16',
                               'CidrBlockState': {'State': 'associated'}}],
  'DhcpOptionsId': 'dopt-831979f9',
  'InstanceTenancy': 'default',
  'IsDefault': False,
  'OwnerId': '033533902081',
  'State': 'available',
  'VpcId': 'vpc-0783e4e5846fbb703'}]

To cut the output I just showed the first few networks that exist in my account. As you can see VPCs have a lot more information associated with them besides the IP address block and the unique identifier. Since this is beyond the scope of this article I’m not going to go over every single one but if you want me I can do it in the comments below so drop me a line. However I do want to explain a few fields here that may be useful for you.

  • State: The state of your VPC is important to know because sometimes it may be disabled or turned off and this gives you visibility to it.
  • Owner: The Identifier of the owner is also good to know for accounting reasons and if you are trying to debug something.

The rest of the options while important I do not believe play a major role for your basic management of your VPC networks using Boto3.

How To Create A VPC With Boto3

Moving on we are going to cover how to add a VPC network using Boto3. Similar to before the logic to this is basically to invoke the function we implemented on our Boto3 VPC function wrapper. The only difference here is that when invoking our function to add a VPC network to our AWS account we are also going to need to pass in the IP address block to it so AWS knows how to allocate it.

import pprint
from boto3_helper import ec2_add_vpc

result = ec2_add_vpc('172.31.0.0/16')
pprint.pprint(result)

In the code above you can see we are basically adding the network IP range with 172.31.0.0 and with a net mask of a class B network (16). This is enough to cover most of your needs generally.

To further demonstrate how this works and see the response that AWS sends back to us when it gets added we will be executing the code to create this new VPC network with the Boto3 code wrapper we implemented.

$ python ./boto3-vpc-add.py
{'CidrBlock': '172.31.0.0/16',
 'CidrBlockAssociationSet': [{'AssociationId': 'vpc-cidr-assoc-077ce7f3e24e9d4db',
                              'CidrBlock': '172.31.0.0/16',
                              'CidrBlockState': {'State': 'associated'}}],
 'DhcpOptionsId': 'dopt-831979f9',
 'InstanceTenancy': 'default',
 'Ipv6CidrBlockAssociationSet': [],
 'IsDefault': False,
 'OwnerId': '033533902081',
 'State': 'pending',
 'VpcId': 'vpc-0c5659f4344044b0a'}
$ python ./boto3-vpc-list.py|grep "vpc-0c5659f4344044b0a"
  'VpcId': 'vpc-0c5659f4344044b0a'},

As you can see above AWS successfully added the VPC network to our list and returned to us the unique VPC identifier to allocated to it, in this case: vpc-0c5659f4344044b0a. As an extra verification we will be leveraging the code to list all VPC networks in our system and basically grep specifically for this new VPC ID to see if it exists in our system. Above it’s visible that the VPC ID exists and it was successfully added to it.

How To Delete VPC With Boto3

In this section we will demonstrate how to delete a VPC network using Boto3. Since we already added a test VPC network earlier we will basically proceed into deleting it now and checking to see if it still exists in the list.

Similarly to the two earlier examples we will use the Boto3 delete VPC wrapper code we wrote and we will be passing to it the VPC ID that we want to mark for deletion.

import pprint
from boto3_helper import ec2_delete_vpc

result = ec2_delete_vpc('vpc-0c170b0963dd0d871')
pprint.pprint(result)

We basically send the command to the AWS service to delete the VPC and then print out the result. The execution of this is shown below.

$ python ./boto3-vpc-delete.py
{'ResponseMetadata': {'HTTPHeaders': {'cache-control': 'no-cache, no-store',
                                      'content-length': '219',
                                      'content-type': 'text/xml;charset=UTF-8',
                                      'date': 'Wed, 27 Jul 2022 18:57:58 GMT',
                                      'server': 'AmazonEC2',
                                      'strict-transport-security': 'max-age=31536000; '
                                                                   'includeSubDomains',
                                      'x-amzn-requestid': 'a860a35a-90e1-4a20-adda-b3959e3d04e1'},
                      'HTTPStatusCode': 200,
                      'RequestId': 'a860a35a-90e1-4a20-adda-b3959e3d04e1',
                      'RetryAttempts': 0}}
$ python ./boto3-vpc-list.py|grep "vpc-0c5659f4344044b0a"
[empty]

As you can see above the command succeeded with a 200 HTTP return code meaning the record of the VPC network we provided was deleted from the AWS account. Since we are adding extra verification we will again check our list for the VPC ID to see if it exists in the list. Running the command shows that the VPC has disappeared and is no longer there indicating we were successfully able to demonstrate how to remove a VPC network using Boto3 from the AWS account.

Conclusion

We were able to successfully go over Python AWS Boto3 VPC Guide, hopefully I answered any questions you may have had and helped you get started on your quest on managing your VPC networks programmatically in your AWS account.

If you found this useful and you think it may have helped you please drop me a cheer below I would appreciate it.

If you have any questions, comments please post them below or send me a note on my twitter. I check periodically and try to answer them in the priority they come in. Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.

Do you prefer to use the AWS console or Boto3 to manage your VPC networks?

I personally still think both have their place and complement each other when working from the terminal or wanting to do customized stuff that require programming and needing to add logic to it.

If you would like to visit the official Python Boto3 documentation here.

If you would like to find more articles related to AWS services:

Leave a Comment

Your email address will not be published. Required fields are marked *