Amazon SQS FIFO Deduplication: Prevent Duplicate Message Processing in .NET

Amazon SQS FIFO queues are designed to enhance messaging between applications when the order of operations and events is critical or where duplicates can't be tolerated.

Even if the same message is sent multiple times to the queue, deduplication ensures that your application processes it only once. This prevents duplicate work and keeps your system reliable.

In this post, let's explore:

• How message deduplication works in SQS FIFO queues
• The different deduplication modes and scopes
• Content-based deduplication vs explicit IDs
• See it all in action using a .NET application

Thanks to AWS for sponsoring this article in my .NET on AWS Series.

Setting Up the Demo Application

The demo application consists of two parts:

• Web API → Publishes weather forecast messages to the SQS FIFO queue
• Console Application → Consumes and processes messages from the queue

To create an SQS FIFO queue, navigate to the AWS Console and under Amazon SQS, select "Create queue". You'll see options for both Standard and FIFO queues.

The Web API has an endpoint that accepts weather data and publishes it to the queue. The key difference from standard queues is the two additional properties required for FIFO queues:

app.MapPost("/weather", async (
    WeatherForecast forecast,
    IAmazonSQS sqsClient) =>
{
    var sendMessageRequest = new SendMessageRequest
    {
        QueueUrl = "your-queue-url",
        MessageBody = JsonSerializer.Serialize(forecast),
        MessageGroupId = forecast.City,
        MessageDeduplicationId = forecast.Id != Guid.Empty
            ? forecast.Id.ToString()
            : null
    };

    await sqsClient.SendMessageAsync(sendMessageRequest);
    return Results.Ok();
});

public record WeatherForecast(
    Guid Id,
    string City,
    WeatherDetails Details);

public record WeatherDetails(
    DateOnly Date,
    int TemperatureC,
    string Summary);

The console application runs a simple processing loop that pulls messages from the SQS queue and logs them to the console:

while (true)
{
    var receiveMessageRequest = new ReceiveMessageRequest
    {
        QueueUrl = "your-queue-url",
        MaxNumberOfMessages = 10
    };

    var response = await sqsClient.ReceiveMessageAsync(receiveMessageRequest);

    foreach (var message in response.Messages)
    {
        Console.WriteLine($"Received: {message.Body}");

        // Delete message after processing
        await sqsClient.DeleteMessageAsync(
            "your-queue-url",
            message.ReceiptHandle);
    }
}

Getting Started with Amazon SQS Standard Queues from .NET

Learn how to get started with Amazon SQS and use it from a .NET Application. We will learn how to send and receive messages, important properties,and concepts that you need to know when using SQS.

rahulpnath.com

Understanding Deduplication in SQS FIFO

Amazon SQS FIFO queues provide multiple ways to handle message deduplication. Let's explore the different approaches and how they work in practice.

MessageDeduplicationId: The 5-Minute Window

The MessageDeduplicationId is a token used in SQS FIFO queues to prevent duplicate message delivery.

Once you set this ID, it ensures that within a 5-minute deduplication window, only one instance of a message with the same ID is processed and delivered.

This means if you accidentally send the exact same message twice, it will only get processed once.

Let's see this in action. Here's a sample message:

{
  "id": "b3b1a1e2-8c2e-4e2a-9c2e-1e2a9c2e1e2b",
  "city": "Brisbane",
  "details": {
    "date": "2025-09-24",
    "temperatureC": 28,
    "summary": "Warm"
  }
}

When I send this message, you can see it gets processed and logged in the console application.

Since we passed an ID, it's being set as the MessageDeduplicationId. If I send the exact same message again within the next 5 minutes, it won't be delivered again.

Even if I send multiple messages with the same ID, they won't get picked up because the deduplication ID is the same for the 5-minute period.

Same ID, Different Content

Here's an important point: when using explicit MessageDeduplicationId, the deduplication is based solely on the ID, not the message content.

Note: SQS FIFO also supports content-based deduplication as an alternative approach, which we'll cover later in this post.

Even if I change the city from Brisbane to Melbourne while keeping the same ID:

{
  "id": "b3b1a1e2-8c2e-4e2a-9c2e-1e2a9c2e1e2b",
  "city": "Melbourne",
  "details": {
    "date": "2025-09-24",
    "temperatureC": 16,
    "summary": "Cool"
  }
}

The message will still be filtered out by SQS. The ID is what matters, not the content or the message group.

To get this message processed, I need to update the ID to a different value. Once I change the ID and send the message again, it gets processed successfully.

SQS FIFO automatically handles all this deduplication for us—no code required on our side.

Deduplication Scope: Queue vs Message Group

In the Amazon SQS FIFO queue settings, there's a configuration called Deduplication Scope with two options:

• Queue (default) → Deduplication applies across the entire queue
• Message Group → Deduplication applies per message group

By default, queues are created with Queue-level deduplication.

With Message Group deduplication enabled, messages are unique per the MessageGroupId we send. In our scenario, we're using the city name as the group ID.

Now, if I send a message for Melbourne with a specific ID, it gets processed successfully. If I then send a message with the same ID but for a different group (city → Brisbane), it also gets processed.

// First message - processed
MessageGroupId = "Melbourne"
MessageDeduplicationId = "b3b1a1e2-8c2e-4e2a-9c2e-1e2a9c2e1e2b"

// Second message - also processed (different group)
MessageGroupId = "Brisbane"
MessageDeduplicationId = "b3b1a1e2-8c2e-4e2a-9c2e-1e2a9c2e1e2b"

This is because with Message Group scope, the deduplication ID is unique per group. The same ID can exist across different message groups.

However, if I send the Brisbane message again with the same ID, it won't get processed because that ID has already been used within that message group's 5-minute window.

This is an important feature that might be useful depending on your application scenario and the number of producers sending messages.

Content-Based Deduplication

SQS FIFO queues also support content-based deduplication. When you enable this feature, SQS automatically generates a deduplication ID based on the message body content.

This means you don't have to send a MessageDeduplicationId explicitly—it becomes optional.

Let's enable content-based deduplication in the queue settings and see how it works.

With content-based deduplication enabled, I'll update the code to send an empty GUID:

MessageDeduplicationId = forecast.Id != Guid.Empty
    ? forecast.Id.ToString()
    : null  // Now sending null

When I send a message to Perth with no deduplication ID:

{
  "id": "00000000-0000-0000-0000-000000000000",
  "city": "Perth",
  "details": {
    "date": "2025-09-24",
    "temperatureC": 32,
    "summary": "Hot"
  }
}

The message gets processed successfully because SQS is seeing this content for the first time.

If I send the exact same message again, it won't get processed. SQS detects that the content is identical and filters it out.

However, if I change even a small detail—let's say the temperature from 32 to 33—the message gets processed again because the content is now different.

Content-Based Deduplication and Message Groups

The deduplication scope setting still applies with content-based deduplication.

With Message Group scope enabled:

• Same content for Perth → Not processed (duplicate within group)
• Same content for Brisbane → Processed (different group)
• Changed content for Brisbane → Processed (new content)

SQS essentially performs a hash of the message body to determine uniqueness. Any change to the content results in a different hash and is treated as a new message within the 5-minute window.

Explicit ID Takes Precedence

Even with content-based deduplication enabled, if you provide a MessageDeduplicationId, it will take precedence over the content-based hashing.

// With explicit ID, content doesn't matter
MessageDeduplicationId = "custom-id-123"

Once you send a message with an explicit ID, that ID is what SQS uses for deduplication, regardless of any content changes.

Keep this in mind when deciding which approach to use in your application.

When to Use Which Deduplication Method

The Amazon SQS documentation provides guidelines on when to use each deduplication approach:

Use Content-Based Deduplication When:

• You have a single producer and single consumer
• Messages are naturally unique because they include an application-specific ID in the body
• The message content itself defines uniqueness

Use Explicit MessageDeduplicationId When:

• You have multiple producers
• Identical message bodies need to be treated as unique messages
• You want explicit control over what constitutes a duplicate
• Messages may be identical but represent different events

The choice between these approaches depends entirely on your application use case and how you're creating messages.

Both options are available on SQS FIFO queues, giving you flexibility in how you handle message deduplication.

👉 Full source code is available here