PynamoDB - Tutorial Python DynamoDB ORM

Table of Contents

Introduction

We will go over a PynamoDB – Tutorial Python DynamoDB ORM.

Did you know that an ORM saves you a lot of typing and lets you use objects instead of queries?

I will break this review in the following sections:

Why using an Python DynamoDB ORM is superior to not using one
Intro on what is PynamoDB and what problem does it solve
How to create, query, update, scan, and get items in DynamoDB using PynamoDB ORM
Test it out our code and see it in action

I have used this successfully in various projects and it works very well and has saved me a ton of time debugging code and database structure.

We will go point by point on getting you up and running in less than 5mins of work.

This is a complete guide and should cover all your questions on PynamoDB – Tutorial Python DynamoDB ORM.

All code and examples on how to do this can be found in the Github link here.

What Is PynamoDB

A lot of people ask me all the time how do I get data from DynamoDB using Python. And the answer is basically always two-fold. If you want the straight forward answer is to use the boto3 library which I have explained in detail in the article below:

Boto3 DynamoDB query, scan, get, put, delete, update items

The answer however is missing one part, is it the best and most efficient way of doing this?

The answer to that is no. Let me explain why, performing queries using the traditional boto3 can be very tedious and require a lot of boiler plate code but more on this later.

The solution to this is basically PynamoDB, a Python DynamoDB ORM that lets you simplify your code interaction with DynamoDB and produce clean code as a result. We will go over why this is advantageous in the following section but for now all you need to do know is that PynamoDB is the library that solves this problem for us and works great.

Why Having An ORM In DynamoDB Python Useful

There are several reasons as to why having a Python DynamoDB ORM is useful, lets go over them in the list below and if that doesn’t compel you then you can continue using the vanilla boto3 client as usual.

Free validation: By default you do not get that but once you strictly define what types you want your models to be you get all the validation and error checking for free from the ORM.
Less Code: The code you end up with is much smaller and easier to read as PynamoDB takes care a lot of the abstraction you would be writing using the boto3 client.
Strict Type Enforcement: Once you define your models you can ensure that the types being sent too will be strictly followed.
Pythonic expressions: PynamoDB follows for the most part python expressions and is based on a class<->table mapping so you can easily navigate all your objects without having to do too much work.
Easy Inheritance: You can inherit easily from your models and take them into other projects once defined.

Let me stress out the fact that in some simple cases and scenarios it may be easier to use boto3 rather than setting up an ORM but as the complexity of your project increases and more fields/tables are being introduced it’s a good idea to make the switch over.

Setting Up The Environment For PynamoDB

Lets start by setting up our environment for PynamoDB to work. Do note that all the tests and everything here are being done locally.

The first thing you need to do is run DynamoDB locally. You can do this by following a guide I wrote here:

How To Test DynamoDB Locally

Once you have DynamoDB running locally you are ready to run some setup steps for PynamoDB.

Effectively we need to do three things here:

Setup a virtual environment to run our code and activate it
Install the package dependencies (they can be found in requirements.txt in the Github repo here)
Ensure PynamoDB was successfully installed.

This process can be seen below.

$ virtualenv venv
created virtual environment CPython3.9.12.final.0-64 in 208ms
  creator CPython3Posix(dest=/Users/alex/code/unbiased/python-pynamodb-orm-tutorial/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/alex/Library/Application Support/virtualenv)
    added seed packages: pip==22.0.4, setuptools==61.2.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

$ source venv/bin/activate

$ pip install -r requirements.txt
Collecting pynamodb
  Downloading pynamodb-5.2.1-py3-none-any.whl (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.8/57.8 KB 1.1 MB/s eta 0:00:00
....
Successfully installed botocore-1.25.0 jmespath-1.0.0 pynamodb-5.2.1 python-dateutil-2.8.2 six-1.16.0 urllib3-1.26.9

$ ipython
In [1]: import pynamodb
In [2]: pynamodb.__version__
Out[2]: '5.2.1'
In [3]: quit()

As you can see we successfully installed PynamoDB version 5.2.1 at the time of this writing. If you have a newer version don’t worry their API seems pretty consistent and your code will still work without many tweaks.

DynamoDB Examples Using PynamoDB

Now that we have PynamoDB setup in our system we are ready to go over some examples that may be useful to cover most of your use-case scenarios on interacting with DynamoDB. If you think I missed something please send me a note and I’ll add an example for that too.

More specifically I’m going to demonstrate the following things:

How to create a DynamoDB model in PynamoDB of your tables
Create / Delete DynamoDB table
Add / Update / Delete / Getting DynamoDB records
Scan and Query examples with conditions for DynamoDB

Define PynamoDB Model Mapping

The first thing we need to do is define a model of the table we will be creating and inserting records in. This follows the example of a previous guide I wrote on testing dynamodb locally which is referenced below.

PynamoDB provides a richest of attributes which define what fields our DynamoDB model will be using. They are all unicode aware and also timezone aware if you are using timestamps. This allows you to have better record tracking of what gets added into your tables.

A sample model table can be outlined below.

from pynamodb.models import Model
from pynamodb.attributes import UnicodeAttribute

class UnbiasedCoderTable(Model):
    class Meta:
        host = 'http://localhost:8000'
        table_name = 'unbiased_coder_table'

    name = UnicodeAttribute(hash_key=True)
    surname = UnicodeAttribute(range_key=True)
    twittername = UnicodeAttribute(null=True)

The code above creates a table called UnbiasedCoderTable that has two keys:

name: This is the hash key also known as partition key
surname: This is the range key also known as sort key

Furthermore it has an optional twitter name attribute which is indicated by null=True allowing you to skip it.

Throughout our examples below we will be referencing this table. One thing to note is that I have specified a host of localhost (this is where the database is running locally). If you are testing directly on AWS you can safely skip that line so it connects to the AWS servers instead of localhost.

The table name is defined in the Meta section of our model and in this case this is how it’s going to be stored in the DynamoDB database. The name we decided to go with is unbiased_coder_table but we will be referencing it throughout this document using the object relation we create called UnbiasedCoderTable.

DynamoDB Create Table Using PynamoDB

Now that we finished defining our table it’s time to write the code that creates it. To do this we will be making use of the create_table function that PynamoDB offers. This code is shown below.

from model import UnbiasedCoderTable

if not UnbiasedCoderTable.exists():
    print ('Unbiased Coder Table does not exist proceed into creating it')
    UnbiasedCoderTable.create_table(read_capacity_units=10, write_capacity_units=10, wait=True)
else:
    print ('Unbiased Coder Table already exists not creating it')

Since we are running locally the read and write capacity set is pretty low values, feel free to adjust whatever you need when going to production. Do note this can also be adjusted later from the AWS console to increase the numbers.

Running the code simply creates the table for us locally which we are going to verify later on by adding a few records into it.

$ python ./pynamodb_create_table.py
Unbiased Coder Table does not exist proceed into creating it

DynamoDB Insert / Add Item Using PynamoDB

So now that we have the table created it’s time that we add a few items to it. Adding items to DynamoDB using PynamoDB is as simple as importing our model and populating the values and then saving the models as shown below.

from model import UnbiasedCoderTable

print ('Adding 3 items in DynamoDB')

unbiased_coder_item1 = UnbiasedCoderTable(
    name = 'Unbiased',
    surname = 'Coder'
).save()

unbiased_coder_item2 = UnbiasedCoderTable(
    name = 'Alex',
    surname = 'Smith'
).save()

unbiased_coder_item3 = UnbiasedCoderTable(
    name = 'John',
    surname = 'Williams'
).save()

print ('Completed saving 3 items in DB')

In the example above we create three records which we will be using through out our examples so we can modify/delete and update them.

If we run the code this will simply create the records which we will verify later on by retrieving them.

$ python ./pynamodb_insert_item.py
Adding 3 items in DynamoDB
Completed saving 3 items in DB

DynamoDB Get Item Using PynamoDB

Now that we have some records added in our database it’s time to test getting a single item. In the example code below we will try to get the first record we added which was: Unbiased Coder.

from model import UnbiasedCoderTable

print ('Getting Item from DynamoDB')

unbiased_coder_item = UnbiasedCoderTable.get('Unbiased', 'Coder')

print ('Retrieved Item: ', unbiased_coder_item.serialize())

Running the example code will verify two things:

The table we previously created exists
The records we added exist and are retrievable

$ python ./pynamodb_get_item.py
Getting Item from DynamoDB
Retrieved Item: {'name': {'S': 'Unbiased'}, 'surname': {'S': 'Coder'}}

As you can see above we successfully retrieved the record we previously added. One thing to note is that PynamoDB offers a shorthand function to serialize and deserialize objects into the AWS DynamoDB JSON format. This is slightly different than traditional JSON formats as it adds an element of type for each field. Since in DynamoDB you do not have special attributes for dates yet those are also considered strings.

However the nice thing about PynamoDB is that it lets you send them as date time stamps enforcing better attribute type checking. In our example we did not have that and simply retrieved the name and surname that we were looking up.

DynamoDB Query/Scan Using PynamoDB

Now that we have the basic functionality of retrieving one item from the database you’d ask what you do if you want to retrieve more and add conditions to it?

The answer is that DynamoDB offers two ways of doing that:

query: Based on an index value or partition/sorting key (pretty fast)
scan: Based purely on conditions (much slower)

The good news is that PynamoDB supports both of those methods pretty well, in fact we will demonstrate both of those executions in the code below.

from model import UnbiasedCoderTable

def print_pynamodb_items(items):
    """
    Prints all PynamoDB Items

    Args:
        items (_type_): List of items
    """
    while True:
        try:
            next_item = items.next()
            print ('Item: ', next_item.serialize())
        except StopIteration:
            break

print ('Getting All Item from DynamoDB Using Scan')

unbiased_coder_items = UnbiasedCoderTable.scan()

print ('Retrieved Item Count Using Scan: ')
print_pynamodb_items(unbiased_coder_items)

unbiased_coder_items = UnbiasedCoderTable.query(hash_key='Unbiased', range_key_condition=UnbiasedCoderTable.surname.startswith('Coder'))

print ('Retrieved Item Count Using Query (name=Unbiased, surname=Coder): ')
print_pynamodb_items(unbiased_coder_items)

Here we are defining a simple scan method without conditions and a query based on the hash and range key we are looking for. The query should return just 1 record since we specified specific criteria for the lookup and the scan in theory should return all of the records that we added earlier in this tutorial (3 of them).

$ python ./pynamodb_query_scan.py
Getting All Item from DynamoDB Using Scan
Retrieved Item Count Using Scan:
Item: {'name': {'S': 'Alex'}, 'surname': {'S': 'Smith'}}
Item: {'name': {'S': 'Unbiased'}, 'surname': {'S': 'Coder'}}
Retrieved Item Count Using Query (name=Unbiased, surname=Coder):
Item: {'name': {'S': 'Unbiased'}, 'surname': {'S': 'Coder'}}

The execution above confirms our observations. One thing to note here is that we wrote a short hand function that basically consumes the iterator that PynamoDB provides to us. Since DynamoDB offers the results in a paginated way the PynamoDB iterator is an elegant abstraction for this to avoid consuming a lot of memory and waiting for the whole query to fetch everything.

The way this works is based on a Last Evaluated Key which is essentially an index of where it was left in order to continue. Again all of this is abstracted from us via the iterator which is really nice. If you were to write the same code in boto3 it would look more complicated even using the latest pagination classes that it provides.

If you are interested into diving deeper in the internals of how this works you can find more information here.

DynamoDB Update Item Using PynamoDB

Now that we have successfully demonstrated retrieving and adding items into DynamoDB we are going to show how we can update them. To do this PynamoDB offers two ways:

Using the update statement
Using the save function as if you were adding a new entry (this is slower and adds a new entry temporarily)

In order to keep things simple we will go with the recommended approach which is using the update statement.

from model import UnbiasedCoderTable

print ('Updating Item name: Unbiased Coder to: Unbiased c0der')

unbiased_coder_item = UnbiasedCoderTable.get('Unbiased', 'Coder')

unbiased_coder_item.update(
    actions=[
        UnbiasedCoderTable.twittername.set('@unbiasedcoder')
    ]
)

In the code above we are basically updating our initial Unbiased Coder record to add a twitter name to it. If you recall earlier in the model definition we had made the twitter name as an optional attribute by specifying null=True.

Let’s go ahead and execute the update syntax and then use the previous code we wrote to retrieve all the records from DynamoDB to see if PynamoDB updated the record correctly.

$ python ./pynamodb_update_item.py
Updating Item name: Unbiased Coder to: Unbiased c0der

$ python ./pynamodb_query_scan.py
Getting All Item from DynamoDB Using Scan
Retrieved Item Count Using Scan:
Item: {'name': {'S': 'Alex'}, 'surname': {'S': 'Smith'}}
Item: {'name': {'S': 'Unbiased'}, 'surname': {'S': 'Coder'}, 'twittername': {'S': '@unbiasedcoder'}}
Retrieved Item Count Using Query (name=Unbiased, surname=Coder):
Item: {'name': {'S': 'Unbiased'}, 'surname': {'S': 'Coder'}, 'twittername': {'S': '@unbiasedcoder'}}

As you can see above both the query and the scan are returning the updated record which includes the previously non-existent twitter name.

DynamoDB Delete Item Using PynamoDB

Finally we will go over deleting a record. Doing this is as simple as retrieving the single record with similar code that we wrote earlier and basically just calling the delete function in that object.

from model import UnbiasedCoderTable

print ('Deleting Item name: John Williams from DynamoDB')

unbiased_coder_item = UnbiasedCoderTable.get('John', 'Williams')

unbiased_coder_item.delete()

To demonstrate this we use the same approach as the update, we run the delete and then we retrieve all the records again to ensure the John Williams name which we are deleting above in the code disappears from our list.

$ python ./pynamodb_delete_item.py
Deleting Item name: John Williams from DynamoDB

$ python ./pynamodb_query_scan.py
Getting All Item from DynamoDB Using Scan
Retrieved Item Count Using Scan:
Item: {'name': {'S': 'Alex'}, 'surname': {'S': 'Smith'}}
Item: {'name': {'S': 'Unbiased'}, 'surname': {'S': 'Coder'}}
Retrieved Item Count Using Query (name=Unbiased, surname=Coder):
Item: {'name': {'S': 'Unbiased'}, 'surname': {'S': 'Coder'}}

As you can see the name is missing from the scan we executed and it’s no longer in the table after the delete script execution.

DynamoDB Delete Table Using PynamoDB

Finally now that we finished demonstrating everything we are going to show how to delete and clean up the table we previously created. Similar to the create_table code we will be leveraging a function called delete_table. This will basically take care of the deletion of our table. You simply need to import your model and call delete_table on it.

from model import UnbiasedCoderTable

if not UnbiasedCoderTable.exists():
    print ('Unbiased Coder Table does not exist theres nothing to delete')
else:
    print ('Found Unbiased Coder Table, proceeding with delete')
    UnbiasedCoderTable.delete_table()

We add an extra check for safety to ensure the table already exists before we issue the delete table command to avoid getting an exception.

$ python ./pynamodb_delete_table.py
Found Unbiased Coder Table, proceeding with delete

And this code successfully has deleted the table. If you would like to verify this you an write a helper script I wrote in a previous article that lists all the available tables in your database. This is available in the article listed below on how to test DynamoDB locally.

Conclusion

We were able to successfully go over PynamoDB – The Ultimate Python DynamoDB ORM, hopefully I answered any questions you may have had and helped you get started on your next DynamoDB project.

If you found this useful and you think it may have helped you please drop me a cheer below I would appreciate it.

If you have any questions, comments please post them below or send me a note on my twitter. I check periodically and try to answer them in the priority they come in. Also if you have any corrections please do let me know and I’ll update the article with new updates or mistakes I did.

Do you currently use an ORM in your DynamoDB code?

I personally like to use an ORM to abstract some of the logic that is library dependent and get for free all the benefits the ORM gives me as discussed earlier in this article.

If you would like to visit the official PynamoDB documentation here.

If you would like to find more articles related to DynamoDB and Databases in general you can check the links below: