BatchWriteItem Operations In DynamoDB Using .NET

Amazon DynamoDB's BatchWriteItem functionality provides a streamlined approach to perform multiple write operations on one or more DynamoDB tables using a single API call.

Using BatchWriteItem you can bundle multiple operations into a single batch,  and significantly reduce the number of API calls and network overhead, leading to improved performance and reduced costs.

BatchWriteItem is particularly useful for scenarios involving data ingestion, bulk updates, or data cleanup tasks, enabling you to efficiently manage changes to your DynamoDB tables while benefiting from DynamoDB's scalable and fully managed infrastructure.

This article is sponsored by AWS and is part of my AWS Series.

In this article, we will learn, how to

• Batch Write Using High-Level API (DynamoDBContext)
• Batch Write Using Low-Level API (AmazonDynamoDBClient)

DynamoDB BatchWriteItem

The BatchWriteItem operation can insert/remove multiple items in one or more DynamoDB tables.

A BatchWriteItem call can send up to 16MB of data over the network, consisting of up to 25 items put or delete operations.

DynamoDB BatchWriteItem can be performed using the High-Level API (using the DynamoDBContext instance) and also the Low-Level API (using the AmazonDynamoDBClient instance).

Let's explore these methods and how to perform the operations on single and multiple tables.

Batch Write Using High-Level .NET API (DynamoDBContext)

The DynamoDBContext is the High-Level API provided in the .NET SDK and abstracts over the Low-Level API. It makes it easier to work with DynamoDB by giving rich .NET classes to interact with.

BatchWriteItem on Single Table Using High-Level API

The DynamoDBContext provides the CreateBatchWrite generic function, that takes in the .NET type representing the DynamoDB Table item. It returns a generic type of BatchWrite.

In the below example, it takes a list of WeatherForecast data, which will be inserted into DynamoDB as a single batch operation.

[HttpPost("batch-write")]
public async Task PostBatch(List<WeatherForecast> batch)
{
    foreach (var data in batch)
        data.Date = data.Date.Date;
    var batchWrite = _dynamoDbContext.CreateBatchWrite<WeatherForecast>();
    batchWrite.AddPutItems(batch);
    await batchWrite.ExecuteAsync();
}

The BatchWrite instance has the AddPutItems method to specify the items to be inserted into the table as part of the batch operation.

Once all items are added to the operation, call the ExecuteAsyncmethod to send data to the DynamoDB table.

BatchWriteItem on Multiple Tables Using High-Level API

Each BatchWrite instance represents a batch operation on a single DynamoDB table. This is specified using the generic type parameter passed to the CreateBatchWrite operation.

The DynamoDBContext uses the table name conventions to determine the table name to perform the queries on.

.NET DynamoDB SDK: Understanding Table Name Conventions and DynamoDBTable Attribute

The DynamoDB .NET SDK by default uses conventions to determine the Table name. However, it supports different options to configure and customize the name. Learn how to achieve this in this post.

Rahul NathRahul Pulikkot Nath

To insert data into multiple tables, we need to create multiple instances of the BatchWrite type passing in the appropriate .NET types.

The below code invokes the CreateBatchWrite on two different types, representing two different tables in DynamoDB.

[HttpPost("batch-write-multiple-tables")]
public async Task PostBatchMulti(BatchWriteData batchItems)
{
    foreach (var data in batchItems.Batch1)
        data.Date = data.Date.Date;
    var batchWrite1 = _dynamoDbContext.CreateBatchWrite<WeatherForecast>();
    batchWrite1.AddPutItems(batchItems.Batch1);

    foreach (var data in batchItems.Batch2)
        data.Date = data.Date.Date;
    var batchWrite2 = _dynamoDbContext
                .CreateBatchWrite<WeatherForecastTable>();
    batchWrite2.AddPutItems(batchItems.Batch2);

    var batches = batchWrite1.Combine(batchWrite2);
    await batches.ExecuteAsync();
}

public class BatchWriteData
{
    public List<WeatherForecast> Batch1 { get; set; }
    public List<WeatherForecastTable> Batch2 { get; set; }    
}

The multiple instances of BatchWrite can then be combined into a single MultiBatchWrite instance using the Conbine method as shown above.

Invoking the ExecuteAsync on the MultiBatchWrite performs both the batch1 and batch2 in a single operation

You can combine any number of BatchWrite instances together as long as it is within the size/count limits.

CreateMultiTableBatchGet Function

The DynamoDBContext also has the function CreateMultiTableBatchWrite that can be used to combine multiple BatchWrite instances.

As shown below, you can pass in the batch instances as a params object into the function to create  the same MultipBatchWrite instance and perform the execute action on it.

var batches = _dynamoDbContext
       .CreateMultiTableBatchWrite(batchWrite1, batchWrite2);
await batches.ExecuteAsync();

Batch Write Using Low-Level .NET API (AmazonDynamoDBClient)

When using the High-Level API it abstracts away a lot of the complexities of the underlying Low-Level API.

However, it also hides some of the functionality that might be useful in certain scenarios.

When the BatchWriteItem requires exceeds the allowed item count/size, it returns a list of UnprocessedKeys collection. This property is available only through the Low-Level API.

Let's see how we can perform the same calls using the Low-Level API

BatchWriteItem on Single Table Using .NET Low-Level API

The AmazonDynamoDBClient provides the BatchWriteAsync function to write items in a batch.

It has multiple overloads and one of them uses the BatchWriteItemRequest as shown below.

[HttpPost("batch-write-client")]
public async Task PostBatchClient(List<WeatherForecast> batch)
{
    foreach (var data in batch)
        data.Date = data.Date.Date;
        
    var batchWriteRequest = new BatchWriteItemRequest
    {
        RequestItems = new Dictionary<string, List<WriteRequest>>
        {
            {
                nameof(WeatherForecast),
                batch.Select(data =>
                    new WriteRequest(new PutRequest(Document
                        .FromJson(JsonSerializer.Serialize(
                            data,
                            new JsonSerializerOptions()
                            {
                                DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
                            }))
                        .ToAttributeMap()))).ToList()
            }
        }
    };
    var response = await _amazonDynamoDbClient
               .BatchWriteItemAsync(batchWriteRequest);
}

The RequestItems takes the collection of items to insert as a WriteRequest object, which again takes a PutRequest instance as shown above.

The list of items is grouped as a dictionary based on the DynamoDB table it is to be written into. The above code writes all the items to the WeatherForecast table.

The BatchWriteItemResponse returns the result of the Batch Write operation. The UnprocessedItems dictionary in the response returns a map of tables and requests against those tables that were not processed.

BatchWriteItem on Multiple Tables Using .NET Low-Level API

To perform a batch write on multiple tables using the .NET Low-Level AmazonDynamoDBClient instance, all we need to do is pass another key value pair of items with the appropriate table name, and the items

var batchWriteRequest = new BatchWriteItemRequest
{
    RequestItems = new Dictionary<string, List<WriteRequest>>
    {
        {nameof(WeatherForecast), <itemsToInsert>},
        {nameof(WeatherForecastTable), <itemsToInsert>}
    }
};

When processing the UnprocessedItems from a batch write response, it's recommended to perform the subsequent request after a random back-off time. You can read more about the recommended practices here.