BatchGetItem Operations In DynamoDB Using .NET

Amazon DynamoDB's BatchGetItem functionality allows you to retrieve multiple items from one or more DynamoDB tables using a single API call.

This feature is useful when retrieving multiple items with known primary keys. It reduces the number of roundtrips to the database and optimizes performance.

In this blog post, let’s learn how to retrieve items in a batch from DynamoDB when building .NET applications.

This article is sponsored by AWS and is part of my AWS Series.

We will learn, how to

• Batch Fetch Using High-Level API (DynamoDBContext)
• Batch Fetch Using Low-Level API (AmazonDynamoDBClient)

DynamoDB BatchGetItem

The BatchGetItem retrieves items from one or more tables in DynamoDB given a set of primary keys.

A single batch operation can retrieve up to 16 MB of data and as many as 100 items in a single roundtrip. If either of these exceeds, the BatchGetItem returns a partial result.

DynamoDB BatchGetItem can be performed using the High-Level API (using the DynamoDBContext instance) and also the Low-Level API (using the AmazonDynamoDBClient instance).

Let's explore these methods and how to perform the operations on single and multiple tables.

Batch Fetch Using HighLevel API (DynamoDBContext)

The DynamoDBContext is the High-Level API provided in the .NET SDK and abstracts over the Low-Level API. It makes it easier to work with DynamoDB by giving rich .NET classes to interact with.

BatchGetItem on Single Table using High-Level API

The DynamoDBContext provides the CreateBatchGet generic function, that takes in the .NET type representing the DynamoDB Table Item. It returns a generic type of BatchGet.

The below API method, taken in a list of primary key values (hash, range key) as a KeyValuePair.

[HttpPost("get-batch-dynamodbcontext-one-table")]
public async Task<List<WeatherForecast>> GetBatch(List<KeyValuePair<string,DateTime>> keys)
{
    var batchGet = _dynamoDbContext.CreateBatchGet<WeatherForecast>();
    foreach (var key in keys)
        batchGet.AddKey(key.Key, key.Value);
    await batchGet.ExecuteAsync();
    return batchGet.Results;
}

You can add the primary keys to retrieve from the specific table using the AddKey method on BatchGet instance.

When you call the ExecuteAsync it performs the batch request to DynamoDB and returns back the results as of the Results property on the BatchGet item as shown above.

BatchGetItem on Multiple Tables using High-Level API

Each BatchGet instance represents a batch operation to one table in DynamoDB. This is specified using the generic type parameter passed to the BatchGet operation.

The DynamoDBContext uses the table name conventions to determine the table name to perform the queries on.

.NET DynamoDB SDK: Understanding Table Name Conventions and DynamoDBTable Attribute

The DynamoDB .NET SDK by default uses conventions to determine the Table name. However, it supports different options to configure and customize the name. Learn how to achieve this in this post.

Rahul NathRahul Pulikkot Nath

To retrieve items from multiple tables, we need to create multiple instances of the BatchGet type using the appropriate .NET Types.

The below code invokes the CreateBatchGet function using two different types, representing two different tables in DynamoDB.

[HttpPost("get-batch-dynamodbcontext-multiple-table")]
public async Task<object> GetBatchMultiTable(List<KeyValuePair<string,DateTime>> keys)
{
    var batchGet1 = _dynamoDbContext.CreateBatchGet<WeatherForecast>();
    var batchGet2 = _dynamoDbContext.CreateBatchGet<WeatherForecastTable>();
    foreach (var key in keys)
    {
        batchGet1.AddKey(key.Key, key.Value);
        batchGet2.AddKey(key.Key, key.Value);
    }

    var batches = batchGet1.Combine(batchGet2);
    await batches.ExecuteAsync();

    return new
    {
        WeatherForecast = batchGet1.Results,
        WeatherForecastTable = batchGet2.Results
    };
}

Once adding the keys to fetch in each of the BatchGet operation, you can combine the multiple BatchGet instances into a single MultiTableBatchGet instance using the Combine method as shown above.

Like before we can perform the ExecuteAsync which invokes the call to DynamoDB to retrieve items from both the BatchGet requests.

To keep things simple, I am getting the same keys from both the tables in the example above, but depending on your application and scenario you will be using different sets of keys and tables to retrieve data.

You can combine any number of BatchGet instances together and perform the call to DynamoDB as a single batch request.

CreateMultiTableBatchGet Function

The DynamoDBContext also has a function CreateMultiTableBatchGet that can be used to combine different BatchGet instances.

Instead of using the Combine method, you can call the CreateMultiTableBatchGet and pass in the BatchGet items as params.

batches = _dynamoDbContext.CreateMultiTableBatchGet(batchGet1, batchGet2);
await batches.ExecuteAsync();

Batch Fetch Using LowLevel API (AmazonDynamoDBClient)

When using the High-Level API it abstracts away a lot of the complexities of the underlying Low-Level API.

However, it also hides some of the functionality that might be useful in certain scenarios.

When the BatchGetItem requires exceeds the allowed item count/size, it returns a list of UnprocessedKeys collection. This property is available only through the Low-Level API.

Let's see how we can perform the same calls using the Low-Level API

BatchGetItem on Single Table using .NET Low-Level API

The AmazonDynamoDBClient provides the BatchGetItemAsync function to get items in a batch.

It has multiple overloads and one of which uses the BatchItemGetRequest as shown below.

The RequestItems takes the list of table names that we are querying from and the keys to get from each of those tables (which is represented using the KeysAndAttributes type).

Below I loop through the keys to fetch from the WeatherForecast table and add the hash and range keys to identify each item to be retrieved from the table.

[HttpPost("get-batch-dynamodbclient-one-table")]
public async Task<List<WeatherForecast>> GetBatchDynamoDbClient(Dictionary<string, DateTime> keys)
{
    var batchGetItemRequest = new BatchGetItemRequest();
    batchGetItemRequest.RequestItems = new Dictionary<string, KeysAndAttributes>()
    {
        {
            nameof(WeatherForecast), new KeysAndAttributes()
            {
                Keys = keys.Select(key => new Dictionary<string, AttributeValue>()
                {
                    {nameof(WeatherForecast.CityName), new AttributeValue(key.Key)},
                    {nameof(WeatherForecast.Date), new AttributeValue(key.Value.ToString(AWSSDKUtils.ISO8601DateFormat))}
                }).ToList()
            }
        }
    };
    var response = await _dynamoDbClient.BatchGetItemAsync(batchGetItemRequest);
    return response.Responses
        .SelectMany(t => t.Value.Select(Document.FromAttributeMap)
        .Select(_dynamoDbContext.FromDocument<WeatherForecast>))
        .ToList();
}

The BatchGetItemAsync function is to perform the fetch from the DynamoDB which returns a BatchGetItemResponse.

The BatchGetItemResponse contains the Responses property which contains the items retrieved in the batch request. It also has the UnprocessedKeys properties which return any keys that weren't processed in the current batch request.

BatchGetItem on Multiple Tables  using .NET Low-Level API

To perform a batch get on multiple tables using the Low-Level AmazonDynamoDBClient instance, all we need to do is pass another key-value pair, indicating the different table name and the associated KeysAndAttributes to retrieve from that table.  

The below request fetches the same items from both WeatherForecast and WeatherForecastTable in DynamoDB.

[HttpPost("get-batch-dynamodbclient-multi-table")]
public async Task<List<WeatherForecast>> GetBatchDynamoDbClientMulti(Dictionary<string, DateTime> keys)
{
    var batchGetItemRequest = new BatchGetItemRequest();
    batchGetItemRequest.RequestItems = new Dictionary<string, KeysAndAttributes>()
    {
        {
            nameof(WeatherForecast), new KeysAndAttributes()
            {
                Keys = keys.Select(key => new Dictionary<string, AttributeValue>()
                {
                    {nameof(WeatherForecast.CityName), new AttributeValue(key.Key)},
                    {nameof(WeatherForecast.Date), new AttributeValue(key.Value.ToString(AWSSDKUtils.ISO8601DateFormat))}
                }).ToList()
            }
        },
        {
            nameof(WeatherForecastTable), new KeysAndAttributes()
            {
                Keys = keys.Select(key => new Dictionary<string, AttributeValue>()
                {
                    {nameof(WeatherForecast.CityName), new AttributeValue(key.Key)},
                    {nameof(WeatherForecast.Date), new AttributeValue(key.Value.ToString(AWSSDKUtils.ISO8601DateFormat))}
                }).ToList()
            }
        }
    };
    var response = await _dynamoDbClient.BatchGetItemAsync(batchGetItemRequest);
    return response.Responses
        .SelectMany(t => t.Value.Select(Document.FromAttributeMap)
            .Select(_dynamoDbContext.FromDocument<WeatherForecast>))
        .ToList();
}

When processing the results you need to make sure to convert the items into the same .NET type based on the table it's reading the items from.

Since in the above case, both items are of the same shape I am converting them to the same type WeatherForecast using the FromDocument method.