Introduction
We will go over a Python AWS Boto3 SNS Guide.
Did you know that SNS supports phone text (sms), email and custom reporting solutions?
We will break down this in the following sections:
- Why it is helpful to use SNS via Python
- How to setup your environment to control AWS SNS
- How to list/create/delete a new topic
- How to subscribe/unsubscribe to a topic and publish a message to it
I have used this successfully in various projects and it works very well and has saved me a ton of time getting going with managing my SNS topics in a programmatic way. We will cover how to do this in Python using the Boto3 library framework.
We will go point by point on getting you up and running in less than 5mins, you do not need to have any programming knowledge to use the tool we are going to be developing here but know Python is preferable if you would like to modify the code for your needs.
This is a complete guide and should cover all your questions on managing load balancers in an AWS environment.
All code and examples on how to do this can be found in the Github link here.
When To Programmatically Control Your AWS SNS
Before I dive into any code and implementation I want to give you a list of reasons of why this may or may not be useful for your needs. Basically whenever programming is involved you need to understand that the nature of your requirements but also capabilities increase so lets go over some reasons for this.
- You can setup conditional SNS topics allowing Python to extend the logic
- You can integrate this in a system that performs devops operations such as a pipeline
- You can do bulk operations which are hard to do manually over the console such as deleting and creating SNS topics
- Your SNS management can be part of a batch job that’s scheduled to run periodically
- You can use this code to automatically check certain things in your AWS account and set filters or rules on how your SNS topics are created
- You can setup monitoring routines for your SNS topics to see if there’s any unusual messages (by being a subscriber to them)
- You can set your own limits on how many SNS topics exist and what is allowed to be exchanged in them
- You can throttle and alert based on how many messages go out
- You can enumerate and ensure the proper policies are enforced and never modified by monitoring them
I’m sure I may have missed some on the lists above and if you do spot something please send me a note and I will be sure to update it.
How To Setup Boto3 To Manage Your SNS Topics
We will start by going over how to setup your system in order to run the Boto3 code which we will be outlining below. Since the Boto3 AWS library is a Python library we will be leveraging here a virtual environment to install our Python libraries to keep the main system clean from any dependencies.
For this I have made the process fairly easy for you by including the needed requirements file which contains all the packages you need to get going. Below you will find an outline of the commands you need to execute to get the environment setup and ready to go. You can also use conda if you prefer that but to keep things simple and lean I’m using directly virtualenv here.
$ virtualenv venv
created virtual environment CPython3.9.12.final.0-64 in 208ms
creator CPython3Posix(dest=/Users/alex/code/unbiased/python-boto3-load-balancer/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/alex/Library/Application Support/virtualenv)
added seed packages: pip==22.1.2, setuptools==62.6.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
$ pip install -r requirements.txt
Collecting boto3
....
Installing collected packages: urllib3, six, python-dotenv, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.24.37 botocore-1.27.37 jmespath-1.0.1 python-dateutil-2.8.2 python-dotenv-0.20.0 s3transfer-0.6.0 six-1.16.0 urllib3-1.26.11
$ python
>>> import dotenv
>>> dotenv.__all__
['get_cli_string', 'load_dotenv', 'dotenv_values', 'get_key', 'set_key', 'unset_key', 'find_dotenv', 'load_ipython_extension']
>>> import boto3
>>> boto3.__version__
'1.24.37'
As you can see from the code above we basically installed two packages:
- dotenv: This contains all the environment variables were will be using in our code such as the AWS access and secret key and also the region of AWS we will be using.
- boto3: This is the AWS library that contains the code we need to connect and interact with our load balancers in the AWS account.
To test the installation we simply run the version and some functions for dotenv to ensure it got installed properly, you can skip that test if you want it’s just a precaution here.
How To Add SNS Policy And Permissions In AWS
The next step is to ensure we have given permissions to our AWS role to be able to interact with the SNS service in AWS. In order to do this we need to navigate into the user we will be modifying to add policy access. Once there we can go ahead and edit the permissions he has and search in the text box under attach existing policies directly for SNS. In this case to keep things simple we will simply give it the full SNS access.
As shown in the screenshot above I have given the user unbiased-coder full SNS permissions in order to be able to interact with it. Note this will fix the following exception if you run into it too:
botocore.errorfactory.AuthorizationErrorException: An error occurred (AuthorizationError) when calling the Subscribe operation: User: arn:aws:iam::XXXX:user/alex is not authorized to perform: SNS:Subscribe on resource: arn:aws:sns:us-east-1:XXXX:unbiased-coder-sns-topic because no resource-based policy allows the SNS:Subscribe action
As shown above this is an access issue regarding the authorization. So it’s basically telling us we are missing the SNS policy permission and the fix for it is that you are missing some SNS permission in this case I was missing, the Subscribe from the policy. Adding it quickly resolved it. Further in this article we will be demonstrating this part and how to subscribe to an SNS topic using Python.
How To Create New SNS Topic
The first thing we will be demonstrating here is how to create a new SNS topic. In order to do this we will be implementing some code using our Boto3 session which you can also find in the Github which is linked below.
Lets analyze the code and see how this works.
import pprint
from boto3_helper import init_aws_session
session = init_aws_session()
sns = session.client('sns')
topic = sns.create_topic(Name='unbiased-coder-sns-topic')
pprint.pprint(topic)
The code works as follows:
- First we import our init session functionality from our helper code that we wrote
- Next we initialize a session to AWS and get an SNS client object
- Finally we create the topic using the sns create_topic function with the name unbiased-coder-sns-topic
To demonstrate this we will be executing the code to create the topic and see what we get back from the AWS server.
$ python boto3-sns-create-topic.py
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '331',
'content-type': 'text/xml',
'date': 'Thu, 22 Sep 2022 09:41:47 GMT',
'x-amzn-requestid': '59b4b14c-f028-5ded-a6d6-eaeac0344a36'},
'HTTPStatusCode': 200,
'RequestId': '59b4b14c-f028-5ded-a6d6-eaeac0344a36',
'RetryAttempts': 0},
'TopicArn': 'arn:aws:sns:us-east-1:XXXX:unbiased-coder-sns-topic'}
The code above receives the response and prints it out as you can see we get back a 200 return code from HTTP which indicates success and also it gives us back a Topic ARN serial which we will use later on to subscribe and unsubscribe to it.
To further verify we can see in the screenshot above that the topic was successfully created. To view this you simply need to navigate in the SNS section of your AWS web console and under that click on topics to see a list of what’s created. However to further verify this we will also be implementing code that lists those topics which is the next part we will be demonstrating.
How To List All SNS Topics
As mentioned above here we will be demonstrating how you can programmatically list all SNS topics in AWS using Python. Similarly to before we will be implementing code that does that using our helper.
import pprint
from boto3_helper import init_aws_session
session = init_aws_session()
sns = session.client('sns')
topics = sns.list_topics()
pprint.pprint(topics)
The code works as follows:
- First we import the helper library and initialize our session as before
- Then we acquire the SNS client object and send a request using list_topics
- Print out the result we get back from the AWS server
If we execute the code we will see the following output:
$ python boto3-sns-list-topics.py
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '389',
'content-type': 'text/xml',
'date': 'Thu, 22 Sep 2022 09:42:09 GMT',
'x-amzn-requestid': 'f2e63e04-6ae6-5a8a-b325-65a25ad706f1'},
'HTTPStatusCode': 200,
'RequestId': 'f2e63e04-6ae6-5a8a-b325-65a25ad706f1',
'RetryAttempts': 0},
'Topics': [{'TopicArn': 'arn:aws:sns:us-east-1:XXXX:unbiased-coder-sns-topic'}]}
As you can above it successfully lists the ARN we previously added. This is indicated by the topic we created which is unbiased-coder-sns-topic and was also listed in the previous section when we created it.
How To Subscribe To An SNS Topic
Now that we have created a topic we will be demonstrating how to subscribe to the SNS topic using Python.
The code we will be implementing is very similar to what we did before.
import pprint
from boto3_helper import init_aws_session
session = init_aws_session()
sns = session.client('sns')
sns_resource = session.resource('sns')
topics = sns.list_topics()
topic_arn = topics['Topics'][0]['TopicArn']
topic = sns_resource.Topic(arn=topic_arn)
subscription = topic.subscribe(Protocol='email', Endpoint='[email protected]', ReturnSubscriptionArn=True)
pprint.pprint(subscription)
Lets examine the code and see how it works:
- First we initialize the session like we did earlier but in this case we also initialize an SNS resource as we will be using it to subscribe
- We then list all of our topics and selectively pick the first one (as we do not have any other ones in the list) if you have more you would need to do a for loop search based on the TopicArn to find which one you need
- After that we need to create a new SNS object from our SNS topic so we can start running actions on it
- The action we will run is basically subscribing to it which we provide as notification method Email along with our address (in this case the endpoint)
- Finally print out the result to see if it was successful.
If we were to execute this we will get back a pending subscription as shown below along with a long unique identifier.
$ python boto3-sns-subscribe-topic.py sns.Subscription(arn='arn:aws:sns:us-east-1:033533902081:unbiased-coder-sns-topic:dcfc788f-494a-4834-844d-6ccb931478b9')
To verify this we check the AWS console to see if it’s listed. Initially it’s pending but once the email is confirmed it goes into Confirmed status and you will start receiving notifications for that topic once you are subscribed.
This is the sample email coming from Amazon AWS to confirm an SNS subscription to a topic.
And finally, once you accept this to finalize the subscription you are taken to the AWS website portal with the confirmation.
This successfully completes our subscription to the SNS topic we previously created.
How To List All SNS Topic Subscriptions
In this part we will be demonstrating how to list all SNS topic subscriptions basically to be able to verify even further that things work but also to get a quick and easy awareness of all the subscribers in your SNS topic.
The code that implements this follows:
from boto3_helper import init_aws_session
session = init_aws_session()
sns = session.client('sns')
sns_resource = session.resource('sns')
topics = sns.list_topics()
topic_arn = topics['Topics'][0]['TopicArn']
topic = sns_resource.Topic(arn=topic_arn)
subscriptions = topic.subscriptions.all()
for subscription in subscriptions:
print('Subscription: ', subscription.arn)
Lets analyze this and see what we did here:
- The first part of the code is similar to before where we acquire a client and an SNS Boto3 resource
- Similarly we list and identify the SNS topic we want
- The next step is to get a list of all the subscriptions specifically for this topic using the all() routine of the subscription object which we previously created.
- Finally we iterate through all of them and print out their ARN number
If we were to execute the code we should be able to see all of our subscriptions for that SNS topic:
$ python boto3-sns-list-subscriptions.py sns.Topic.subscriptionsCollection(sns.Topic(arn='arn:aws:sns:us-east-1:033533902081:unbiased-coder-sns-topic'), sns.Subscription)
As shown above we simply have one the one we previously created.
How To Publish SNS Message For Lambda Function
Now that we have an SNS topic created and an active subscription to it to start getting notification we will see how we can post a notification to it like a message via a Lambda function.
This code snipet below can be used wherever you want in your Lambda or separate script code:
import pprint
from boto3_helper import init_aws_session
session = init_aws_session()
sns = session.client('sns')
sns_resource = session.resource('sns')
topics = sns.list_topics()
topic_arn = topics['Topics'][0]['TopicArn']
topic = sns_resource.Topic(arn=topic_arn)
result = topic.publish(Message='Hello from Unbiased Coder')
pprint.pprint(result)
- The first step is what we described before to find the topic ARN and create an object out of it
- Once this is created and we have the topic we can go ahead and use the publish function to send a message to our topic which all subscribers will receive. In this case we will be broadcasting to it: Hello from Unbiased Coder
- Finally we print out the result to see if the message successfully posted
If we are to execute this we see the output below:
$ python boto3-sns-publish-message.py
{'MessageId': '986dfdfb-a291-5ae3-9546-a65dd0603774',
'ResponseMetadata': {'HTTPHeaders': {'content-length': '294',
'content-type': 'text/xml',
'date': 'Thu, 22 Sep 2022 14:26:57 GMT',
'x-amzn-requestid': '67f6078a-e82f-54f6-a2ad-5c2c9ade2961'},
'HTTPStatusCode': 200,
'RequestId': '67f6078a-e82f-54f6-a2ad-5c2c9ade2961',
'RetryAttempts': 0}}
As you can see execution was successful and we basically sent a message to our topic. As a consequence since we previously subscribed to get email notifications from it we will be checking our email client to see if something came through in the form as shown below.
This email basically confirms that our subscription works and we got the message posted to it by our publish function.
How To Unsubscribe From SNS Topic
Finally if you want to do some cleanup or dynamically control subscriptions in your topic you can use the unsubscribe function. The code for this is shown below:
import pprint
from boto3_helper import init_aws_session
session = init_aws_session()
sns = session.client('sns')
sns_resource = session.resource('sns')
topics = sns.list_topics()
topic_arn = topics['Topics'][0]['TopicArn']
topic = sns_resource.Topic(arn=topic_arn)
subscriptions = topic.subscriptions.all()
for subscription in subscriptions:
result = subscription.delete()
pprint.pprint(result)
Lets go over quickly the new steps that we have here:
- The initialization is similar to the subscription where we find and create a subscription object.
- The next step is to simply basically call delete on that and it will remove it from AWS
- Finally we print out the result to see if it worked
And if we go ahead and execute it as shown below:
$ python boto3-sns-unsubscribe-topic.py
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '201',
'content-type': 'text/xml',
'date': 'Thu, 22 Sep 2022 14:19:00 GMT',
'x-amzn-requestid': 'e2935828-2e90-5054-9f60-038e5f8403de'},
'HTTPStatusCode': 200,
'RequestId': 'e2935828-2e90-5054-9f60-038e5f8403de',
'RetryAttempts': 0}}
We see the 200 response which basically indicates the subscription to that SNS topic was successfully removed.
How To Delete SNS Topic
Finally we need to do a bit more cleaning up to delete the SNS topic we previously created. In this part of the code we will demonstrate how you can implement deletion of an SNS topic using Python.
The code for this is shown below:
import pprint
from boto3_helper import init_aws_session
session = init_aws_session()
sns = session.client('sns')
topic = sns.delete_topic(TopicArn='arn:aws:sns:us-east-1:033533902081:unbiased-coder-sns-topic')
pprint.pprint(topic)
- This works similarly to what we did for our creation of the topic we import our helper and get an SNS helper object for the client
- Then we can simply pass on the topic ARN to the delete_topic function and the topic will be deleted
- Finally we print out the result to ensure things worked
If we are to execute the code above we will see the following output:
$ python boto3-sns-delete-topic.py
{'ResponseMetadata': {'HTTPHeaders': {'content-length': '201',
'content-type': 'text/xml',
'date': 'Thu, 22 Sep 2022 14:28:24 GMT',
'x-amzn-requestid': '8a38b426-d6a0-554e-9209-ef80a5e99cca'},
'HTTPStatusCode': 200,
'RequestId': '8a38b426-d6a0-554e-9209-ef80a5e99cca',
'RetryAttempts': 0}}
As you can see above the SNS topic was successfully deleted as we have a 200 response code in HTTP.
To further confirm this we check the topic listing under the SNS AWS web console page to see if something is there.
As shown above the topic no longer exists which means it has been deleted from the system successfully.
Conclusion
We were able to successfully go over a Python AWS Boto3 SNS Guide, hopefully I answered any questions you may have had and helped you get started on your quest on managing your SNS programmatically in your AWS account.
If you found this useful and you think it may have helped you please drop me a cheer below I would appreciate it.
If you have any questions, comments please post them below or send me a note on my twitter. I check periodically and try to answer them in the priority they come in. Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.
Do you prefer to use the AWS console or Boto3 to manage SNS?
I personally still think both have their place and complement each other when working from the terminal or wanting to do customized stuff that require programming and needing to add logic to it. SNS is a great tool and offers great potential if used correctly and programmatic actions make it even more powerful.
If you would like to find more articles related to AWS services:
- Python AWS Boto3 Load Balancer Guide
- How To Create An AWS Python Lambda Layer
- Github Repo for this code
You can find some references to what we discussed earlier here: